三斗室
2016-12-31T09:02:45+08:00
http://chenlinux.com/
陈子
rao.chenlin@gmail.com
2016 年度个人总结
2016-12-30T00:00:00+08:00
http://chenlinux.com/2016/12/30/report-of-this-year
<p>老习惯,一年年底给自己做一个总结。</p>
<h2 id="section">写作</h2>
<p>今年博客写得越发的少了,只有十一篇。其中还有两篇是思辨类的文字,只有九篇是技术笔记。但是这么写并不代表我在检讨自己,因为这是我工作八年来第一次转变自己的角色,不再是一个单纯的运维,或者架构师,而是『日志易』的产品经理。甚至放大一点说,作为目前国内最领先的『日志分析工具』的唯一的产品经理,思考 IT 人员到底需要的是什么样的东西,才是我最需要做的事情。</p>
<p>这两篇思考,一篇是<a href="http://chenlinux.com/2016/11/15/important-unuseful-feature-in-log-analysis/">日志分析中 6 个常见但没啥用的功能</a>,一篇是<a href="http://chenlinux.com/2016/03/19/machine-vs-ops/">机器战胜人类了,伺候机器的运维呢?</a>。看起来我是个很喜欢唱反调的 PM 呢……</p>
<p>所以年终总结上必须给可能还剩下的博客读者们证明一下自己,请一定阅读下面这篇演讲稿:<a href="https://sway.com/xpkkQ2ifSS7D8CTa">海量数据驱动的智能运维</a>。</p>
<p>这篇演讲是为了 velocity 大会创作的。今年外出演讲依然不少,但是大多是企业内训或者行业会议,公开的技术大会最终只参加了这场,而且又接近年尾,可以说整篇演讲算是集个人全年思考之大成:</p>
<ul>
<li>运维和机器之间要什么样的交互方式?</li>
<li>运维的知识库如何自然而然的积累和继承?</li>
<li>机器学习到底能怎么用到运维上?</li>
<li>复杂环境下日志标准应该怎么定?</li>
</ul>
<p>不过这次演讲实际上并没有受到很大的欢迎,在后来联系 DBAplus 社群想在推一波的时候还被拒了,理由是太水了没干货。联想一下 2015 年曾经广受欢迎的那些演讲,我总结一下什么是大众希望的干货呢:我们现在跑了 100 台的集群,10 TB 的数据;用到了 spark、kafka、docker;一开始这不行那不行,后来一看,哦,有个参数要改一下;架构图如下;性能指标监控截屏如下;我们下一步打算再上 100 TB……</p>
<h2 id="elk-">ELK 社区</h2>
<p>虽然在做一个商业化产品,但是紧跟 ELK 开源社区依然是我个人爱好没变。</p>
<p>3 月份,借安快创业谷的场地办了一次小型的 ELK 用户 meetup,形式很随意,我自己给大家演示了一下 juttle 项目,请京东的 LiuYuBao 分享了一下他们踩的坑。最后要求在场所有人必须至少发言一次,说心得说感想说废话均可。这个要求直接导致 meetup 肯定是小型的,事实上到场的也就是 20 人的规模,效果还是不错的。本来还打算请滴滴的 taowen 分享一下他的 es-monitor 项目,这样就可以再发起一次 meetup,不过失败了,所以最终也就只办了这么一次。</p>
<p>5 月份,把之前博客上的 kibana server plugin 整理了一下,发到了 GitHub 上,取了个名字叫 <a href="https://github.com/chenryn/kaae">KaaE</a>。核心思想就是模仿 watcher 项目的配置语法实现 Kibana 里运行的告警监控,这样可以节省写自己文档的时间——直接让用户看 watcher 的官方文档就好了。到 7 月的时候,因为在 GitHub 上时不时吆喝,lmangani 童鞋也加入进来一起开发了。lmangani 是曾经另一个 kibana3 fork(qbana) 的作者,也是经验丰富而且脑洞不小,在我的 server plugin 基础上加上了 spy plugin,让用户可以直接在 Kibana 的 Visualize 界面上点击保存 watcher 条件!这真是一个天才的设计!随后我们一致认为官方的 report 做的逻辑太绕,又给 KaaE 加上了报表功能。可以自负的说,KaaE 比官方的 watcher 和 report 都好用的多。</p>
<p>10 月份,lmangani 加入了 SIREn 公司,KaaE 改名叫做 sentinL,以后将作为 kibi 的一个插件继续开发。我再次拒绝了 SIREn 的邀请(第一次是我写 <a href="https://github.com/chenryn/kbn_sankey_vis">kbn_sankey_vis</a> 插件的时候),不过倒很乐意 KaaE 项目换一种形式继续焕发活力,lmangani 加油!</p>
<p>接着是 Elastic 中国开发者大会,提交了一个话题想讲讲 KaaE 的开发。不过被拒了,大抵上还是小插件的开发不太受欢迎吧。于是很欢快的和朋友们在台下一边听演讲一边交(tu)流(cao),场面非常热闹,ELK 大势所趋,当初 wood 叔预测的 ES admin 职位肯定不久就会诞生了~</p>
<p>12 月份,针对 ELK 5.0 版本的文档基本修改完毕,交给出版社校对,修订稿有 185 页,相当于第一版页数的一半了。或许在春节后可以面世。</p>
<h2 id="section-1">翻译</h2>
<p>今年做了两件翻译事,不幸都 happy ending。一件是 ES 中文社区组织翻译《Elasticsearch 权威指南》,忝列 D 组组长,然而说实话,要组织十来个网友按时干活,难度比在公司里组织同时干活难多了。人就不理你 QQ 消息,你能如何?都想翻译不想 review,你也没办法……</p>
<p>另一件是《Learning Puppet 4》,原计划 10 月就应该交稿。不过连着碰到意外情况,到现在还有三章没完成。希望春节前可以努把力……</p>
<h2 id="section-2">工作</h2>
<p>产品经理确实是一项非常有意思的工作。刚开始免不了茫然,年初我曾经满网络的搜寻各种产品经理入门啊,产品经理必读啊的资料。后来反应过来:第一、这些 2C 的资料对我目前没什么用;第二、这些零零碎碎的资料压根也不适合真的入门而适合吹牛;第三、我需要的是扬长避短。</p>
<p>这里也要感谢研发童鞋,他们耐心的等到我招来了专职的交互设计师以后,才笑告我:『你的 Axure 画的真烂!』</p>
<p>这一整年,能在开源基础上折腾的花样,心中有数,手下也基本做的差不多。盘点一下心中的计划,对未来我还是信心满满的。明年,我们肯定要玩个大的~</p>
<h2 id="section-3">生活</h2>
<p>总结的最后,才是最重要的:接下来的这个春节是我人生最期待的一个春节了。我的『小渔』就要到来!迫不及待的心情啊~~小渔,欢迎你</p>
日志分析中 6 个常见但没啥用的功能
2016-11-15T00:00:00+08:00
http://chenlinux.com/2016/11/15/important-unuseful-feature-in-log-analysis
<p>日志分析是 IT 运维领域非常重要的一部分工作。甚至可以说,在平台化、模块化、服务化盛行的今天,这部分工作的重要性已经逼近传统的设备监控。不过日志由于来源、使用者、管理者都比设备指标要复杂,导致日志分析的功能需求,也庞大很多。在这些庞大的,或者说『泥沙俱下』的功能需求中,有那么一些然并卵的,或许因为听起来很炫酷,或许因为想延续过去的使用习惯,今天因为出差到外地,难得有空放松下,决定吐槽几个这种然并卵的功能。</p>
<h2 id="realtime-alert">realtime alert</h2>
<p>排在第一位的就是所谓的『实时告警』。做一个告警系统,其实可以分成两类不同的目的:</p>
<ul>
<li>出现了问题要修复,</li>
<li>快要出问题得避免。</li>
</ul>
<p>那么分开说:</p>
<p>如果是要喊人来修复的,假设你的告警内容已经细化到完全不用再排查问题,从告警发出来,到你登录到服务器解决问题,至少也需要数分钟级别 —— 根据墨菲定律,这时候你很可能在睡觉在吃饭在坐车在团建,那么十分钟已经是你行动迅速了。那么告警是第 0.1 秒发出来的,跟是第 10 秒发出来的,有什么区别?而把告警从间隔 10 秒压缩到 1 秒内的实时,需要花费的架构调整和成本上升,可不是一点半点……(你说一个关键字实时过滤没啥成本?那你需要先加强一下告警系统的追踪、扩展、抑制等功能呢,告警没那么简单)</p>
<p>如果是要提前避免的,一般你的基础架构已经进化的不错了,才会想要通过告警的触发动作来自动化修改你的流量、资源和任务调度编排。这种需求其实更多归入容量规划范畴,很难想象这种事情要实时性干嘛,谁家平台不打余量的?</p>
<p>当然,不管上面哪种,我吐槽的都是追求 1 秒甚至毫秒的实时。如果你的监控间隔还停留在 5 分钟以上,可别拿我这段话做挡箭牌 —— 如果你从收到告警到解决问题需要小时级别,5 分钟可能是也不算多,但是你的故障定位方式,或者说告警系统的内容细化水平,更加需要提高。</p>
<h2 id="section">翻页翻页翻页</h2>
<p>排在第二位的就是 show me more money,错了,logline。日志分析系统一般都会在界面上列出来日志原文供查看。而一帮『手贱』的人,就会很 happy 地点下一页下一页下一页下~一~页~下~然后系统出问题了。</p>
<p>这个功能需求其实就是过去 <code class="highlighter-rouge">cat logfile | grep KEYWORD | less</code> 习惯的遗毒。上来就恨不得自己能 vim 进去一行行开始看日志。Ctrl+F 嗷嗷翻页固然很爽,不知不觉中时间全都浪费掉了 —— 想想上一条你还想要的『实时』 —— 运维排查问题最适合的思路是快速试错!一个想法验证下不行赶紧验证下一个。如果一页 20 条日志你看不出来,两页 40 条日志你看不出来,你就赶紧改个时间段、改个关键词吧。</p>
<p>当然,话说回来,老想着往后翻页,也有可能是真想不出来改用啥关键词。日志分析系统有必要提供帮助用户更快找到合适关键词的能力。这东西就是仪表盘可视化。利用正确的能力做正确的事,而不应该在有正确的方法的情况下继续使用麻烦办法。</p>
<h2 id="section-1">经纬度地图</h2>
<p>既然说到可视化,可视化方面是做日志分析乃至数据分析最容易误入歧途的方向了。有兴趣的可以看下面几个链接,是我从 Kibana Plugin 社区讨论组里复制过来的:</p>
<ul>
<li><a href="http://www.businessinsider.com/the-27-worst-charts-of-all-time-2013-6?op=1&IR=T">http://www.businessinsider.com/the-27-worst-charts-of-all-time-2013-6?op=1&IR=T</a></li>
<li><a href="http://flowingdata.com/category/visualization/ugly-visualization/">http://flowingdata.com/category/visualization/ugly-visualization/</a></li>
</ul>
<p>这些很复杂的可视化就不提了。在日志分析方面,最常见的一个炫酷的效果就是地图。地图可真是一个被各种玩出花来的东西,诸如安全攻击喜欢放个 3D 地球,在 google 图片上随便搜『DDoS atack earth』关键词,大把大把;做个推广活动,喜欢搞个实时连线的中国地图看 PV,全国各地,来一个访问,飞一个点出来到北京。。。</p>
<p>真的是酷毙了。不过,然后呢?你看到这个点能干嘛?而且飞动中的点,唰唰就过去了,压根捕捉不到。</p>
<p>说到实际情况,IT 日志分析需要地图的大多数时候是基于行政区划的统计。全局负载均衡绝大多数都是以行政区划和运营商为基准做的划分,如果通过地图真的定位到什么访问问题,很大可能下一步你能做的是通过商务手段去联系当地电信服务运营商!你要经纬度有什么用?—— 别忘了免费的 GeoIP 国内精准度本来就低。花点时间搞一个准确到地市运营商的 IP 地址库,才是最应该做的事情。</p>
<h2 id="etl-to-bi">全量下载(etl to BI)</h2>
<p>另一个和翻页有些类似的功能,就是要求全量日志下载。这种需求通常目的也是分两类,一类其实跟翻页是一个需求,不知道查啥内容,干脆要求把日志都下载回来自己慢慢折腾;另一类则是环境中有一些标准的 BI 软件,觉得日志分析软件的可视化和统计方法不够用,还是喜欢、习惯 BI,所以要求日志分析系统负责搜索,BI 系统负责分析。</p>
<p>这块怎么说呢,列出来有些个人主观化,我个人不太觉得在 IT 运维领域,有啥是 BI 能做,而开源日志分析项目做不来的事情。退一步说:真要两个系统的结合,也应该是分层的架构。充分利用日志分析系统的分布式架构并行处理能力,将大量 map 操作在日志系统完成,将中间统计结果导入到 BI 中完成最后的 reduce 工作即可。</p>
<p>非要把原日志(即使是归一化之后的结构数据)导入到 BI 里做统计,是一个耗时耗力的下下之选。</p>
<h2 id="sql">SQL</h2>
<p>第四个很常见的功能,就是 SQL。这甚至不是日志分析领域的毛病,在所有和数据相关的、非关系型数据库的数据存储系统上,都会有大把人问:有 SQL 支持么?</p>
<p>就我的浅薄见识,对所有存储系统要 FUSE 挂载,对所有数据系统要 SQL 查询,应该是可以对等的两个吃力不讨好的工作了。在 Hadoop 上有无数个实现 SQL 的项目,哪怕 Hive 和 SparkSQL 这种级别的大项目在,我还是要说:研发同仁们想要 SQL,不就是觉得自己已经会 SQL,所以要无缝对接,不用学习新知识么?你们点开 Hive 文档,里面有多少是非标准 SQL 的函数功能?</p>
<p>只有极少数基础的、简单的过滤和统计函数,可以横跨 API、SQL、DSL 等方式,在各平台上都通用。而你选择某个大数据平台的实际理由,大多是它的xxx yyy zzz亮点功能,很好,你需要自己搞一个 UDF 了……这还搞 SQL 有什么意义。</p>
<p>从编程语言学来一个经验,对特定领域,采用特定领域语言,即 DSL 的设计方式,永远是更加高效、灵活、优秀的选择。</p>
<p>在日志分析方面来说,抓住关键词检索、分组统计、上下文关联、时间序列这几个特性,你就可以抽象出来几个能覆盖足够场景的函数了,而借鉴命令行操作的形式,从左到右的书写习惯也比 SQL 的从右到左的形式更加符合数据流向的效果。</p>
<p>熟悉日志分析领域的人可能看出来我是在给 SPL 写软文了……自 Splunk 发明 SPL 这种日志分析领域的 DSL 以来,已经有大批日志分析产品跟进了这个形式,SumoLogic、Rizhiyi、XpoLog、MicroSoft Azure、Oracle Cloud Management 等等。不过公平的说,上面一段要点,确实也可以提炼出来跟 SPL 不一样的 DSL 设计,比如说:更接近面向对象编程语言的链式调用函数,同样也符合这个习惯 —— 这也是 ELK 从 5.0 开始分发的 timelion 插件的选择。</p>
<h2 id="live-tail">live tail</h2>
<p>今天我能想到的最后一个恶习遗毒,同样还符合酷炫概念的功能,是 live tail,也有叫 web tail 或者 log tail 的。不知道从哪来的程序员情节,觉得终端的黑底白字最棒了,非要在浏览器页面上,通过 websocket 连接上某台服务器,实时查看某个日志文件的尾部滚动。或者简单说,就是一个 <code class="highlighter-rouge">tail -F logfile</code> 功能的网页化。</p>
<p>由于网络的限制、浏览器渲染的限制(毕竟要很多酷炫效果呢),这类功能一般实现出来带有诸多的限制:</p>
<ul>
<li>直接从 agent 建联,意味着后续的归一化结构是无法用来做复杂过滤的,同样还意味着跨平台能力削弱;</li>
<li>需要限制使用者的并发数,以及每个连接的流速。一般来说是每秒不许超过 1000 条 —— 人肉眼其实每秒也看不过来这么多数据;</li>
<li>为了限速,必须指定具体的 hostname 和 filename,无法使用通配符,无法跨文件关联查询;</li>
<li>为了解决跨文件,在同一页面上切分屏幕,考虑美观和视觉,最多也就是切分一次,即一次可以看两个文件的 tail。</li>
</ul>
<p>我在最前面已经说到了,日志系统之所以现在重要性提高,就是因为日志前所未有的分散,两个分屏的 tail,有什么用?</p>
<p>当然,回到这个伪需求的根本目的:我就是在调试而不是事后排错呢,怎么让我可以快速看到我横跨好几个模块的调试日志是否正常?</p>
<p>这跟前面『无限翻页』类似:你真正需要的知道新入的日志有没有异常,而不是刷过去什么字样。通过 AND OR NOT 等过滤条件,通过时间排序,通过关联 ID,你完全可以在秒级得到更精准的、更有利于你阅读的日志。</p>
<hr />
<p>就写到这里吧,我犹豫很久要不要把人工智能机器学习写进来。考虑到异常探测和预测也算是机器学习的一部分,还是不一竿子打倒全部吧~~这里只说一句:我花时间翻了一打 IT 运维日志相关的机器学习论文,用神经网络的效果普遍比回归差。嗯~总之,大家老实干活就好了。</p>
Elastic 官方压测工具 rally 试用
2016-08-19T00:00:00+08:00
testing
elasticsearch
http://chenlinux.com/2016/08/19/es-rally
<p>rally 工具是 Elastic 官方开源的针对性性能压测工具。目前 Elasticsearch 的 nightly performance report 就是由 rally 产生的。对自己在做 ES 源码修改,或者ES 应用调优的人来说,通过 rally 验证自己的修改效果,是一件很需要且容易的事情。</p>
<p>rally 依赖 python3.4+,所以为了试用直接在自己电脑上安装比较快。直接 <code class="highlighter-rouge">pip3 install esrally</code> 即可。</p>
<p>电脑上没有 gradle 的无法从最新 master 代码编译(Macbook 上即使通过 dmg 安装的 gradle 也识别不到)。只能下 binary 包。所以运行方式为:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>/opt/local/Library/Frameworks/Python.framework/Versions/3.5/bin/esrally --pipeline=from-distribution --distribution-version=1.7.3
</code></pre>
</div>
<p>默认情况下压测采用的数据集叫 geonames,是一个 2.8GB 大的 JSON 数据。ES 也提供了一系列其他类型的压测数据集。如果要切换数据集采用 <code class="highlighter-rouge">--track</code> 参数:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>/opt/local/Library/Frameworks/Python.framework/Versions/3.5/bin/esrally --pipeline=from-distribution --distribution-version=1.7.3 --track=geonames
</code></pre>
</div>
<p>重复运行的时候可以修改 ~/.rally/rally.ini 里的 <code class="highlighter-rouge">tracks[default.url]</code> 为第一次运行时下载的地址:<strong>~/.rally/benchmarks/tracks/default</strong> 。然后离线运行:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>/opt/local/Library/Frameworks/Python.framework/Versions/3.5/bin/esrally --offline --pipeline=from-distribution --distribution-version=1.7.3 --track=geonames
</code></pre>
</div>
<p>静静等待程序运行完毕,就会给出一个漂亮的输出结果了。</p>
<p>这个运行会是一个很漫长的时间,如果你其实只关心部分的性能,比如只关心写入,不关心搜索。其实可以自己去修改一下 track 的任务定义。</p>
<p>track 的定义文件在 <code class="highlighter-rouge">~/.rally/benchmarks/tracks/default/geonames/track.json</code>。如果你改动较大,建议直接新建一个 track 目录,比如叫 <code class="highlighter-rouge">mytest/track.json</code> 。</p>
<p>对照 geonames 里的定义,有各种 operations,然后在 challenges 里指明调用哪些 operation。最后运行命令的时候通过 <code class="highlighter-rouge">--challenge=</code> 参数来指定执行哪个。</p>
<p>下面是一段我在本机采用默认压测数据集 geonames 的结果:</p>
<table>
<thead>
<tr>
<th>version</th>
<th>eps</th>
<th>index size</th>
</tr>
</thead>
<tbody>
<tr>
<td>1.7.3</td>
<td>12650</td>
<td>2.67GB</td>
</tr>
<tr>
<td>2.3.2</td>
<td>10344</td>
<td>3.31GB</td>
</tr>
<tr>
<td>5.0.0-alpha2</td>
<td>11903</td>
<td>3.19GB</td>
</tr>
</tbody>
</table>
<p>差距好大啊?!然后我发现 1.7.3 用的 mapping 没加 doc_values,修改 <code class="highlighter-rouge">~/.rally/benchmarks/tracks/default/geonames/mappings.json</code> ,都加上后重新测试结果:</p>
<p>10448eps 3.25GB</p>
<p>接着再关闭 <code class="highlighter-rouge">_all</code> 结果:</p>
<p>12630eps 2.73GB</p>
<p>接着再关闭<code class="highlighter-rouge">_field_names</code> 结果:</p>
<p>14662eps 2.71GB</p>
<p>以及打开<code class="highlighter-rouge">_field_names</code> 关闭 <code class="highlighter-rouge">_source</code> 结果:</p>
<p>13121eps 2.04GB</p>
<p>在关闭<code class="highlighter-rouge">_all</code>和<code class="highlighter-rouge">_field_names</code>的基础上,mapping中分词字符串字段加上</p>
<div class="highlighter-rouge"><pre class="highlight"><code>"index_options": "docs",
"norms": {
"enabled": false
}
</code></pre>
</div>
<p>定义的结果:</p>
<p>16226eps 2.6GB</p>
<p>写入速度大概提高了10%。</p>
<p>如果要用自己的数据集呢,也一样是在自己的 track.json 里定义,比如:</p>
<pre><code class="language-JSON">{
"meta": {
"data-url": "/Users/raochenlin/.rally/benchmarks/data/splunklog/1468766825_10.json.bz2"
},
"indices": [
{
"name": "geonames",
"types": [
{
"name": "type",
"mapping": "mappings.json",
"documents": "1468766825_10.json.bz2",
"document-count": 924645,
"compressed-bytes": 19149532,
"uncompressed-bytes": 938012996
}
]
}
],
</code></pre>
<p>这里就是用的一份 splunkd 的 internal 日志,JSON 导出。日志原长度为 166152239,导出 JSON 长度为 938012996。</p>
<p>同样做一次写入压测,结果为:</p>
<ul>
<li>关闭<code class="highlighter-rouge">_field_names</code>:7193.5eps,索引大小358.173MB。</li>
<li>关闭<code class="highlighter-rouge">_field_names</code>和norms:8216.5eps,345.536MB。</li>
<li>关闭<code class="highlighter-rouge">_source</code>和norms:6615eps,192.817MB。</li>
</ul>
elasticsearch 的 sampler 聚合
2016-07-21T00:00:00+08:00
elasticsearch
elasticsearch
http://chenlinux.com/2016/07/21/sampler-aggregation
<p>在上一篇文章的基础上,其实 Elasticsearch 从 2.0 以后,还新增了另一种聚合方式,叫 sampler。这个聚合的作用,是在每个分片上,只采样部分文档出来继续后续统计。</p>
<p>比如把上一篇的查询改成这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
curl -XPOST <span class="s1">'localhost:9200/logstash-2016.07.18/logs/_search?pretty&terminate_after=10000&size=0'</span> -d <span class="s1">'
{
"aggs": {
"group": {
"terms": {
"field": "result.punct"
},
"aggs": {
"sample": {
"sampler": {
"shard_size": 200
},
"aggs": {
"keyword": {
"significant_terms": {
"size": 1,
"field": "result._raw"
},
"aggs": {
"hit": {
"top_hits": {
"_source": {
"include": [ "result._raw" ]
},
"size":1
}
}
}
}
}
}
}
}
}
}
'</span>
</code></pre>
</div>
<p>当然,在这个 raw 日志的情况下,取样意义不是特别到,因为有 <code class="highlighter-rouge">terminate_after</code> 在,采样本身不会绝对随机。但是对其他 <code class="highlighter-rouge">doc_values</code> 的字段,采样就有意义了。</p>
山寨一个 Splunk 的事件模式功能
2016-07-18T00:00:00+08:00
logstash
elasticsearch
splunk
http://chenlinux.com/2016/07/18/event-pattern
<p>之前我曾经讲过一个简单的在 ELK 中山寨 Splunk 的『显示来源』功能的办法。这次我们玩个更有难度的、当然依然只是山寨式功能的新东西:『事件模式』功能。</p>
<p>Splunk 6.2 推出的这个功能,会基于当前搜索语句的结果集做模式探测,根据精度调整,做成不同数量的聚类。然后给每个聚类分组内,提取出一个关键词(个别情况下也有零个或多个的)。也就是通过机器学习的手段,探测你的日志可能有什么模式,其最具识别性的关键内容是什么。</p>
<p><img src="/images/uploads/splunk-event-pattern.png" alt="Event Pattern" /></p>
<p>这个页面如果用 SPL 表示,就是:<code class="highlighter-rouge">index=_internal | cluster t=0.8 lableonly=true | findkeywords labelfield=cluster_label | sort - percentInputGroup</code></p>
<p><img src="/images/uploads/splunk-findkeywords.png" alt="findkeywords command" /></p>
<p>我们目前当然在 ES 里是没法做聚类分析什么的了。不过在日志场景下,也不是没有近似的办法。</p>
<h3 id="section">第一步:完成山寨版的日志模式分组</h3>
<p>其实如何山寨模式分组,Splunk 也有类似 SPL 命令做出了示范。这个命令叫 <code class="highlighter-rouge">typelearner</code>。</p>
<p>这个命令的大致意思是:把日志里的英文单词、数字、空格等字符都隐藏掉,剩下各种标点符号,就代表一种日志类型。简单的处理方式就是:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>cat samplelog.cisco.asa |sed <span class="s1">'s/[0-9a-zA-Z]*//g'</span> | sed <span class="s1">'s/[[:space:]]/_/g'</span>
</code></pre>
</div>
<p>然后将这个纯标点符号的字符串,存为事件的一个字段,我们沿袭 Splunk 的叫法: <strong>punct</strong> 。</p>
<p>这样,我们只要简单的对 punct 字段做 <code class="highlighter-rouge">terms aggregation</code> 就可以获取模式分组了。</p>
<h3 id="section-1">第二步:完成分组内的关键词查找</h3>
<p>然后查找关键词。什么叫关键词呢?就是要能让本分组跟其他分组有显著差异的一个词。这个显然不能再用 terms aggregation 了。否则出来的是最多的词,而不是最有差异性的词。ES 对这个也提供了现成的聚合方式:<code class="highlighter-rouge">significant_terms aggregation</code>。</p>
<p>然后这里有另一个问题:一般我们都是在 <code class="highlighter-rouge">not_analyzed</code> 字段上做聚合统计的。现在显然并没有具体哪个字段来提供单个字段值做聚合!我们需要用的就是<strong>分词的日志原文内容</strong>。</p>
<p>所以这块我们需要对原文字段的 mapping 做出特殊定义:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="w"> </span><span class="s2">"message"</span><span class="err">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"text"</span><span class="p">,</span><span class="w">
</span><span class="nt">"fielddata"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="nt">"index_options"</span><span class="p">:</span><span class="w"> </span><span class="s2">"docs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"norms"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span><span class="p">}</span><span class="err">,</span><span class="w">
</span></code></pre>
</div>
<p>即重新放开 fielddata —— ES 5.0 里,text 类型字段已经默认关闭 fielddata 了。</p>
<p>至于内存的问题,或者交给 Circuit Breaker 来控制;或者自己通过请求中的 <code class="highlighter-rouge">terminate_after</code> 参数预先控制。</p>
<p>就模式发现这个功能来说,通过 <code class="highlighter-rouge">terminate_after</code> 参数预定义控制应该是个不错的思路。因为本来就是一个不确定的猜测,加太大的数据量来做这事儿,没多少性价比。</p>
<p>所以我们最终发出的请求是这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
curl -XPOST <span class="s1">'http://localhost:9200/logstash-2016.07.18/logs/_search?pretty&terminate_after=30000&size=0'</span> -d <span class="s1">'
{
"aggs": {
"group": {
"terms": {
"field": "punct"
},
"aggs": {
"keyword": {
"significant_terms": {
"size": 1,
"field": "message"
},
"aggs": {
"hit": {
"top_hits": {
"_source": {
"include": [ "message" ]
},
"size":1
}
}
}
}
}
}
}
}
'</span>
</code></pre>
</div>
<p>我们可以看到请求结果如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nt">"took"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">2179</span><span class="p">,</span><span class="w">
</span><span class="nt">"timed_out"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nt">"terminated_early"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nt">"_shards"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="p">,</span><span class="w">
</span><span class="nt">"successful"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w">
</span><span class="nt">"failed"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w">
</span><span class="nt">"failures"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"shard"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w">
</span><span class="nt">"index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2016.07.18"</span><span class="p">,</span><span class="w">
</span><span class="nt">"node"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"L0qQ1ZcyQGmj7Ge7ZlCmYg"</span><span class="p">,</span><span class="w">
</span><span class="nt">"reason"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"circuit_breaking_exception"</span><span class="p">,</span><span class="w">
</span><span class="nt">"reason"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"[request] Data too large, data for [<reused_arrays>] would be larger than limit of [415550668/396.2mb]"</span><span class="p">,</span><span class="w">
</span><span class="nt">"bytes_wanted"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">415762160</span><span class="p">,</span><span class="w">
</span><span class="nt">"bytes_limit"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">415550668</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">371095</span><span class="p">,</span><span class="w">
</span><span class="nt">"max_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">0.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"aggregations"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"group"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"doc_count_error_upper_bound"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">72</span><span class="p">,</span><span class="w">
</span><span class="nt">"sum_other_doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">93355</span><span class="p">,</span><span class="w">
</span><span class="nt">"buckets"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"--_::._+____-_=,_=,_=,_=.,_=,_="</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">98100</span><span class="p">,</span><span class="w">
</span><span class="nt">"keyword"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">98100</span><span class="p">,</span><span class="w">
</span><span class="nt">"buckets"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"cpu_seconds"</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">98100</span><span class="p">,</span><span class="w">
</span><span class="nt">"score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">2.2037623779471813</span><span class="p">,</span><span class="w">
</span><span class="nt">"bg_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">115831</span><span class="p">,</span><span class="w">
</span><span class="nt">"hit"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">98100</span><span class="p">,</span><span class="w">
</span><span class="nt">"max_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2016.07.18"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AVX-RMJbLjo3PexoUujh"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"_source"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"07-15-2016 14:17:33.776 +0800 INFO Metrics - group=pipeline, name=indexerpipe, processor=index_thruput, cpu_seconds=0.000000, executes=111, cumulative_hits=161675"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"--_::._+____-_=,_=,_=,_=,_=,_=,_="</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">87058</span><span class="p">,</span><span class="w">
</span><span class="nt">"keyword"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">87058</span><span class="p">,</span><span class="w">
</span><span class="nt">"buckets"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"largest_size"</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">75663</span><span class="p">,</span><span class="w">
</span><span class="nt">"score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">2.835574761742766</span><span class="p">,</span><span class="w">
</span><span class="nt">"bg_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">75663</span><span class="p">,</span><span class="w">
</span><span class="nt">"hit"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">75663</span><span class="p">,</span><span class="w">
</span><span class="nt">"max_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2016.07.18"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AVX-RMJbLjo3PexoUuj9"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"_source"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"07-15-2016 14:17:02.780 +0800 INFO Metrics - group=queue, name=nullqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"--_::._+____-_=,_=\"\",_=.,_=.,_=.,_=,_=.,_="</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">26317</span><span class="p">,</span><span class="w">
</span><span class="nt">"keyword"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">26317</span><span class="p">,</span><span class="w">
</span><span class="nt">"buckets"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"max_age"</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">26317</span><span class="p">,</span><span class="w">
</span><span class="nt">"score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">7.224805514306611</span><span class="p">,</span><span class="w">
</span><span class="nt">"bg_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">45119</span><span class="p">,</span><span class="w">
</span><span class="nt">"hit"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">26317</span><span class="p">,</span><span class="w">
</span><span class="nt">"max_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2016.07.18"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AVX-RMJbLjo3PexoUukH"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"_source"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"07-15-2016 14:17:02.780 +0800 INFO Metrics - group=per_sourcetype_thruput, series=\"scheduler\", kbps=0.014869, eps=0.032258, kb=0.460938, ev=1, avg_age=0.000000, max_age=0"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"--_::._+____-_=,_=\"//////.\",_=.,_=.,_=.,_=,_=.,_="</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">13063</span><span class="p">,</span><span class="w">
</span><span class="nt">"keyword"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">13063</span><span class="p">,</span><span class="w">
</span><span class="nt">"buckets"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"log"</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">13063</span><span class="p">,</span><span class="w">
</span><span class="nt">"score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">27.241628614916287</span><span class="p">,</span><span class="w">
</span><span class="nt">"bg_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">13140</span><span class="p">,</span><span class="w">
</span><span class="nt">"hit"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">13063</span><span class="p">,</span><span class="w">
</span><span class="nt">"max_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2016.07.18"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AVX-RMKILjo3PexoUulQ"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"_source"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"07-15-2016 14:16:31.780 +0800 INFO Metrics - group=per_source_thruput, series=\"/applications/splunk/var/log/splunk/metrics.log\", kbps=0.326188, eps=2.032164, kb=10.112305, ev=63, avg_age=0.968254, max_age=1"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"--_::._+____-_=,_=,_=.,_=.,_=.,_=.,_=.,_=."</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">11603</span><span class="p">,</span><span class="w">
</span><span class="nt">"keyword"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">11603</span><span class="p">,</span><span class="w">
</span><span class="nt">"buckets"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"average_kbps"</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">11603</span><span class="p">,</span><span class="w">
</span><span class="nt">"score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">20.38013481592441</span><span class="p">,</span><span class="w">
</span><span class="nt">"bg_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">17357</span><span class="p">,</span><span class="w">
</span><span class="nt">"hit"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">11603</span><span class="p">,</span><span class="w">
</span><span class="nt">"max_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2016.07.18"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AVX-RMKILjo3PexoUulA"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"_source"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"07-15-2016 14:16:31.781 +0800 INFO Metrics - group=thruput, name=index_thruput, instantaneous_kbps=0.875684, instantaneous_eps=2.032165, average_kbps=0.340430, total_k_processed=33138.000000, kb=27.147461, ev=63.000000"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"--_::._+____-_=,_=,_=,_=,_="</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">11417</span><span class="p">,</span><span class="w">
</span><span class="nt">"keyword"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">11417</span><span class="p">,</span><span class="w">
</span><span class="nt">"buckets"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"qwork_units"</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">11417</span><span class="p">,</span><span class="w">
</span><span class="nt">"score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">31.50372251905054</span><span class="p">,</span><span class="w">
</span><span class="nt">"bg_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">11417</span><span class="p">,</span><span class="w">
</span><span class="nt">"hit"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">11417</span><span class="p">,</span><span class="w">
</span><span class="nt">"max_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2016.07.18"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AVX-RMLOLjo3PexoUunn"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"_source"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"07-15-2016 14:15:29.777 +0800 INFO Metrics - group=tpool, name=indexertpool, qsize=0, workers=2, qwork_units=0"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"--_::._+____-_=,_=,_=---,_=.,_=,_="</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">11350</span><span class="p">,</span><span class="w">
</span><span class="nt">"keyword"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">11350</span><span class="p">,</span><span class="w">
</span><span class="nt">"buckets"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"generic"</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">11350</span><span class="p">,</span><span class="w">
</span><span class="nt">"score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">31.69559471365639</span><span class="p">,</span><span class="w">
</span><span class="nt">"bg_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">11350</span><span class="p">,</span><span class="w">
</span><span class="nt">"hit"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">11350</span><span class="p">,</span><span class="w">
</span><span class="nt">"max_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2016.07.18"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AVX-RMJbLjo3PexoUukk"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"_source"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"07-15-2016 14:17:02.779 +0800 INFO Metrics - group=pipeline, name=indexerpipe, processor=syslog-output-generic-processor, cpu_seconds=0.000000, executes=104, cumulative_hits=161564"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"--_::._+____-_=,_=,_=,_=."</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">7135</span><span class="p">,</span><span class="w">
</span><span class="nt">"keyword"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">7135</span><span class="p">,</span><span class="w">
</span><span class="nt">"buckets"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"search_health_metrics"</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">7135</span><span class="p">,</span><span class="w">
</span><span class="nt">"score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">51.010511562718996</span><span class="p">,</span><span class="w">
</span><span class="nt">"bg_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">7135</span><span class="p">,</span><span class="w">
</span><span class="nt">"hit"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">7135</span><span class="p">,</span><span class="w">
</span><span class="nt">"max_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2016.07.18"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AVX-RMJbLjo3PexoUujq"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"_source"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"07-15-2016 14:17:33.776 +0800 INFO Metrics - group=search_health_metrics, name=bundle_directory_reaper, bundle_dir_reaper_max_ms=1, bundle_dir_reaper_mean_ms=1.000000"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"--_::._+____-_=,_=,_=,_=.,_=,_=,_=,_="</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">5849</span><span class="p">,</span><span class="w">
</span><span class="nt">"keyword"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">5849</span><span class="p">,</span><span class="w">
</span><span class="nt">"buckets"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"search_queue_metrics"</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">5849</span><span class="p">,</span><span class="w">
</span><span class="nt">"score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">62.445888186014706</span><span class="p">,</span><span class="w">
</span><span class="nt">"bg_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">5849</span><span class="p">,</span><span class="w">
</span><span class="nt">"hit"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">5849</span><span class="p">,</span><span class="w">
</span><span class="nt">"max_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2016.07.18"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AVX-RMKILjo3PexoUulx"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"_source"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"07-15-2016 14:16:31.777 +0800 INFO Metrics - group=search_concurrency, name=search_queue_metrics, enqueue_seaches_count=0, avg_time_spent_in_queue=0.000000, max_time_spent_in_queue=0, current_queue_size=0, largest_queue_size=0, min_queue_size=0"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"--_::._+____-_=,_=,_=,_=,_=,_=,_=,_=,_=,_=,_=,_=,_"</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">5848</span><span class="p">,</span><span class="w">
</span><span class="nt">"keyword"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">5848</span><span class="p">,</span><span class="w">
</span><span class="nt">"buckets"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"max_ready"</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">5848</span><span class="p">,</span><span class="w">
</span><span class="nt">"score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">62.45673734610123</span><span class="p">,</span><span class="w">
</span><span class="nt">"bg_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">5848</span><span class="p">,</span><span class="w">
</span><span class="nt">"hit"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">5848</span><span class="p">,</span><span class="w">
</span><span class="nt">"max_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2016.07.18"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AVX-RMJbLjo3PexoUuk5"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"_source"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"07-15-2016 14:17:02.776 +0800 INFO Metrics - group=searchscheduler, dispatched=1, skipped=0, total_lag=1, max_ready=0, max_pending=0, max_lag=1, window_max_lag=0, window_total_lag=0, max_running=0, actions_triggered=0, completed=1, total_runtime=0.189, max_runtime=0.189"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
</div>
<p>响应体中可以看到因为 <code class="highlighter-rouge">terminate_after</code> 设得还是过大,所以还没到中止条数就被 kill 了。实际只扫描了 370173 条数据。那么我们下次就可以把 <code class="highlighter-rouge">terminate_after</code> 调成 10000 得了。</p>
<p>然后就是 <code class="highlighter-rouge">significant_terms</code> 返回的关键词们。跟之前 splunk 的截图相比,我们可以发现,不是完全一样的效果,但是还是有部分关键词是一致的。比如 <code class="highlighter-rouge">smallest_size</code>, <code class="highlighter-rouge">total_k_processed</code>, <code class="highlighter-rouge">search_health_metrics</code>, <code class="highlighter-rouge">var</code>, <code class="highlighter-rouge">workers</code> 等。</p>
<p>可以说,作为一个山寨品,这个做法是行得通的~</p>
hapi.js 框架的认证授权插件示例
2016-07-07T00:00:00+08:00
logstash
javascript
kibana
http://chenlinux.com/2016/07/07/hapi-auth
<p>Kibana 4.x 在服务器端采用了 hapi.js 框架开发。虽然目前依然没有认证和授权的插件出来(官方 Kibana 的 shield 插件应该只是做了一个认证,授权部分是由 ES 本身的 shield 插件完成的)。不过既然叫框架嘛,自然就是有不少扩展可用。本文简要介绍一下 hapi.js 框架的认证授权插件的用法。有兴趣的读者可以自己稍微改造一下,就能让 Kibana 也有认证授权功能了。</p>
<p>首先准备一下环境:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>mkdir hapi-auth-simple
cd hapi-auth-simple
npm init
npm install --save bcrypt
npm install --save hapi
npm install --save hapi-rbac
npm install --save hapi-auth-cookie
</code></pre>
</div>
<p>你就会发现目录底下多出来一个 <code class="highlighter-rouge">node_modules/</code> 目录和 <code class="highlighter-rouge">package.json</code> 配置定义文件。定义如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hapi-auth-test"</span><span class="p">,</span><span class="w">
</span><span class="nt">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1.0.0"</span><span class="p">,</span><span class="w">
</span><span class="nt">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span><span class="w">
</span><span class="nt">"main"</span><span class="p">:</span><span class="w"> </span><span class="s2">"index.js"</span><span class="p">,</span><span class="w">
</span><span class="nt">"scripts"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"test"</span><span class="p">:</span><span class="w"> </span><span class="s2">"echo \"Error: no test specified\" && exit 1"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"author"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span><span class="w">
</span><span class="nt">"license"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ISC"</span><span class="p">,</span><span class="w">
</span><span class="nt">"dependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"bcrypt"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^0.8.7"</span><span class="p">,</span><span class="w">
</span><span class="nt">"hapi"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^13.5.0"</span><span class="p">,</span><span class="w">
</span><span class="nt">"hapi-auth-cookie"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^6.1.1"</span><span class="p">,</span><span class="w">
</span><span class="nt">"hapi-rbac"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^2.2.0"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
</div>
<p>然后开始写实际的 demo 代码啦。<code class="highlighter-rouge">index.js</code> 内容如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="s1">'use strict'</span><span class="p">;</span>
<span class="kr">const</span> <span class="nx">Bcrypt</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'bcrypt'</span><span class="p">);</span>
<span class="kr">const</span> <span class="nx">Hapi</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'hapi'</span><span class="p">);</span>
<span class="kr">const</span> <span class="nx">Rbac</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'hapi-rbac'</span><span class="p">);</span>
<span class="kr">const</span> <span class="nx">Cookie</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'hapi-auth-cookie'</span><span class="p">);</span>
<span class="kr">const</span> <span class="nx">server</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Hapi</span><span class="p">.</span><span class="nx">Server</span><span class="p">();</span>
<span class="nx">server</span><span class="p">.</span><span class="nx">connection</span><span class="p">({</span> <span class="na">port</span><span class="p">:</span> <span class="mi">3000</span> <span class="p">});</span>
<span class="kd">let</span> <span class="nx">uuid</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="kr">const</span> <span class="nx">users</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">john</span><span class="p">:</span> <span class="p">{</span>
<span class="na">username</span><span class="p">:</span> <span class="s1">'john'</span><span class="p">,</span>
<span class="na">password</span><span class="p">:</span> <span class="s1">'$2a$10$iqJSHD.BGr0E2IxQwYgJmeP3NvhPrXAeLSaGCj6IR/XU5QtjVu5Tm'</span><span class="p">,</span> <span class="c1">// 'secret'</span>
<span class="na">name</span><span class="p">:</span> <span class="s1">'John Doe'</span><span class="p">,</span>
<span class="na">group</span><span class="p">:</span> <span class="p">[</span><span class="s1">'user'</span><span class="p">]</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="kr">const</span> <span class="nx">login</span> <span class="o">=</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">request</span><span class="p">,</span> <span class="nx">reply</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">request</span><span class="p">.</span><span class="nx">auth</span><span class="p">.</span><span class="nx">isAuthenticated</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">reply</span><span class="p">.</span><span class="nx">redirect</span><span class="p">(</span><span class="s1">'/'</span><span class="p">);</span>
<span class="p">}</span>
<span class="kd">let</span> <span class="nx">message</span> <span class="o">=</span> <span class="s1">''</span><span class="p">;</span>
<span class="kd">let</span> <span class="nx">account</span> <span class="o">=</span> <span class="kc">null</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">request</span><span class="p">.</span><span class="nx">method</span> <span class="o">===</span> <span class="s1">'post'</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">request</span><span class="p">.</span><span class="nx">payload</span><span class="p">.</span><span class="nx">username</span> <span class="o">||</span>
<span class="o">!</span><span class="nx">request</span><span class="p">.</span><span class="nx">payload</span><span class="p">.</span><span class="nx">password</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">message</span> <span class="o">=</span> <span class="s1">'Missing username or password'</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">else</span> <span class="p">{</span>
<span class="nx">account</span> <span class="o">=</span> <span class="nx">users</span><span class="p">[</span><span class="nx">request</span><span class="p">.</span><span class="nx">payload</span><span class="p">.</span><span class="nx">username</span><span class="p">];</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">account</span> <span class="o">||</span>
<span class="o">!</span><span class="nx">Bcrypt</span><span class="p">.</span><span class="nx">compareSync</span><span class="p">(</span><span class="nx">request</span><span class="p">.</span><span class="nx">payload</span><span class="p">.</span><span class="nx">password</span><span class="p">,</span> <span class="nx">account</span><span class="p">.</span><span class="nx">password</span><span class="p">))</span> <span class="p">{</span>
<span class="nx">message</span> <span class="o">=</span> <span class="s1">'Invalid username or password'</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">request</span><span class="p">.</span><span class="nx">method</span> <span class="o">===</span> <span class="s1">'get'</span> <span class="o">||</span> <span class="nx">message</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">reply</span><span class="p">(</span><span class="s1">'<html><head><title>Login page</title></head><body>'</span> <span class="o">+</span>
<span class="p">(</span><span class="nx">message</span> <span class="p">?</span> <span class="s1">'<h3>'</span> <span class="o">+</span> <span class="nx">message</span> <span class="o">+</span> <span class="s1">'</h3><br/>'</span> <span class="p">:</span> <span class="s1">''</span><span class="p">)</span> <span class="o">+</span>
<span class="s1">'<form method="post" action="/login">'</span> <span class="o">+</span>
<span class="s1">'Username: <input type="text" name="username"><br>'</span> <span class="o">+</span>
<span class="s1">'Password: <input type="password" name="password"><br/>'</span> <span class="o">+</span>
<span class="s1">'<input type="submit" value="Login"></form></body></html>'</span><span class="p">);</span>
<span class="p">}</span>
<span class="kr">const</span> <span class="nx">sid</span> <span class="o">=</span> <span class="nb">String</span><span class="p">(</span><span class="o">++</span><span class="nx">uuid</span><span class="p">);</span>
<span class="nx">request</span><span class="p">.</span><span class="nx">server</span><span class="p">.</span><span class="nx">app</span><span class="p">.</span><span class="nx">cache</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="nx">sid</span><span class="p">,</span> <span class="p">{</span> <span class="na">account</span><span class="p">:</span> <span class="nx">account</span> <span class="p">},</span> <span class="mi">0</span><span class="p">,</span> <span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">reply</span><span class="p">(</span><span class="nx">err</span><span class="p">);</span>
<span class="p">}</span>
<span class="nx">request</span><span class="p">.</span><span class="nx">cookieAuth</span><span class="p">.</span><span class="nx">set</span><span class="p">({</span> <span class="na">sid</span><span class="p">:</span> <span class="nx">sid</span> <span class="p">});</span>
<span class="k">return</span> <span class="nx">reply</span><span class="p">.</span><span class="nx">redirect</span><span class="p">(</span><span class="s1">'/'</span><span class="p">);</span>
<span class="p">});</span>
<span class="p">};</span>
<span class="nx">server</span><span class="p">.</span><span class="nx">register</span><span class="p">([</span><span class="nx">Cookie</span><span class="p">,</span> <span class="nx">Rbac</span><span class="p">],</span> <span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="p">{</span>
<span class="k">throw</span> <span class="nx">err</span><span class="p">;</span>
<span class="p">}</span>
<span class="kr">const</span> <span class="nx">cache</span> <span class="o">=</span> <span class="nx">server</span><span class="p">.</span><span class="nx">cache</span><span class="p">({</span>
<span class="na">segment</span><span class="p">:</span> <span class="s1">'sessions'</span><span class="p">,</span>
<span class="na">expiresIn</span><span class="p">:</span> <span class="mi">3</span> <span class="o">*</span> <span class="mi">24</span> <span class="o">*</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">1000</span>
<span class="p">});</span>
<span class="nx">server</span><span class="p">.</span><span class="nx">app</span><span class="p">.</span><span class="nx">cache</span> <span class="o">=</span> <span class="nx">cache</span><span class="p">;</span>
<span class="nx">server</span><span class="p">.</span><span class="nx">auth</span><span class="p">.</span><span class="nx">strategy</span><span class="p">(</span><span class="s1">'session'</span><span class="p">,</span> <span class="s1">'cookie'</span><span class="p">,</span> <span class="s1">'required'</span><span class="p">,</span> <span class="p">{</span>
<span class="na">password</span><span class="p">:</span> <span class="s1">'password-should-be-32-characters'</span><span class="p">,</span>
<span class="na">cookie</span><span class="p">:</span> <span class="s1">'sid-example'</span><span class="p">,</span>
<span class="na">redirectTo</span><span class="p">:</span> <span class="s1">'/login'</span><span class="p">,</span>
<span class="na">isSecure</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="na">validateFunc</span><span class="p">:</span> <span class="p">(</span><span class="nx">request</span><span class="p">,</span> <span class="nx">session</span><span class="p">,</span> <span class="nx">callback</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">cache</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="nx">session</span><span class="p">.</span><span class="nx">sid</span><span class="p">,</span> <span class="p">(</span><span class="nx">err</span><span class="p">,</span> <span class="nx">cached</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">callback</span><span class="p">(</span><span class="nx">err</span><span class="p">,</span> <span class="kc">false</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">cached</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">callback</span><span class="p">(</span><span class="kc">null</span><span class="p">,</span> <span class="kc">false</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nx">callback</span><span class="p">(</span><span class="kc">null</span><span class="p">,</span> <span class="kc">true</span><span class="p">,</span> <span class="nx">cached</span><span class="p">.</span><span class="nx">account</span><span class="p">);</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="nx">server</span><span class="p">.</span><span class="nx">route</span><span class="p">([</span>
<span class="p">{</span>
<span class="na">method</span><span class="p">:</span> <span class="p">[</span><span class="s1">'GET'</span><span class="p">,</span> <span class="s1">'POST'</span><span class="p">],</span>
<span class="na">path</span><span class="p">:</span> <span class="s1">'/login'</span><span class="p">,</span>
<span class="na">config</span><span class="p">:</span> <span class="p">{</span>
<span class="na">handler</span><span class="p">:</span> <span class="nx">login</span><span class="p">,</span>
<span class="na">auth</span><span class="p">:</span> <span class="p">{</span> <span class="na">mode</span><span class="p">:</span> <span class="s1">'try'</span> <span class="p">},</span>
<span class="na">plugins</span><span class="p">:</span> <span class="p">{</span>
<span class="s1">'hapi-auth-cookie'</span><span class="p">:</span> <span class="p">{</span>
<span class="nl">redirectTo</span><span class="p">:</span> <span class="kc">false</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="na">method</span><span class="p">:</span> <span class="s1">'GET'</span><span class="p">,</span>
<span class="na">path</span><span class="p">:</span> <span class="s1">'/logout'</span><span class="p">,</span>
<span class="na">config</span><span class="p">:</span> <span class="p">{</span>
<span class="na">handler</span><span class="p">:</span> <span class="p">(</span><span class="nx">request</span><span class="p">,</span> <span class="nx">reply</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">request</span><span class="p">.</span><span class="nx">cookieAuth</span><span class="p">.</span><span class="nx">clear</span><span class="p">();</span>
<span class="k">return</span> <span class="nx">reply</span><span class="p">.</span><span class="nx">redirect</span><span class="p">(</span><span class="s1">'/'</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="na">method</span><span class="p">:</span> <span class="s1">'GET'</span><span class="p">,</span>
<span class="na">path</span><span class="p">:</span> <span class="s1">'/'</span><span class="p">,</span>
<span class="na">config</span><span class="p">:</span> <span class="p">{</span>
<span class="na">handler</span><span class="p">:</span> <span class="p">(</span><span class="nx">request</span><span class="p">,</span> <span class="nx">reply</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">reply</span><span class="p">(</span><span class="s1">'<html><head></head><body>Welcome: '</span> <span class="o">+</span>
<span class="nx">request</span><span class="p">.</span><span class="nx">auth</span><span class="p">.</span><span class="nx">credentials</span><span class="p">.</span><span class="nx">name</span> <span class="o">+</span>
<span class="s1">'<form method="get" action="/logout">'</span> <span class="o">+</span>
<span class="s1">'<input type="submit" value="Logout">'</span> <span class="o">+</span>
<span class="s1">'</form></body></html>'</span><span class="p">);</span>
<span class="p">},</span>
<span class="na">plugins</span><span class="p">:</span> <span class="p">{</span>
<span class="na">rbac</span><span class="p">:</span> <span class="p">{</span>
<span class="na">target</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s1">'credentials:group'</span><span class="p">:</span> <span class="s1">'user'</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s1">'credentials:group'</span><span class="p">:</span> <span class="s1">'admin'</span>
<span class="p">}</span>
<span class="p">],</span>
<span class="na">apply</span><span class="p">:</span> <span class="s1">'permit-overrides'</span><span class="p">,</span>
<span class="na">policies</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="na">target</span><span class="p">:</span> <span class="p">{</span>
<span class="s1">'credentials:group'</span><span class="p">:</span> <span class="s1">'admin'</span>
<span class="p">},</span>
<span class="na">effect</span><span class="p">:</span> <span class="s1">'permit'</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="na">target</span><span class="p">:</span> <span class="p">{</span>
<span class="s1">'credentials:group'</span><span class="p">:</span> <span class="s1">'user'</span>
<span class="p">},</span>
<span class="na">apply</span><span class="p">:</span> <span class="s1">'permit-overrides'</span><span class="p">,</span>
<span class="na">rules</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="na">target</span><span class="p">:</span> <span class="p">{</span>
<span class="s1">'credentials:username'</span><span class="p">:</span> <span class="s1">'john'</span><span class="p">,</span>
<span class="p">},</span>
<span class="na">effect</span><span class="p">:</span> <span class="s1">'permit'</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="na">effect</span><span class="p">:</span> <span class="s1">'deny'</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">]);</span>
<span class="nx">server</span><span class="p">.</span><span class="nx">start</span><span class="p">((</span><span class="nx">err</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="p">{</span>
<span class="k">throw</span> <span class="nx">err</span><span class="p">;</span>
<span class="p">}</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'server running at: '</span> <span class="o">+</span> <span class="nx">server</span><span class="p">.</span><span class="nx">info</span><span class="p">.</span><span class="nx">uri</span><span class="p">);</span>
<span class="p">});</span>
<span class="p">});</span>
</code></pre>
</div>
<p>就这样,一个简单的认证授权页就完成了。运行 <code class="highlighter-rouge">node index.js</code> 命令,打开浏览器,输入 <code class="highlighter-rouge">127.0.0.1:3000</code> 即可验证效果。</p>
<p>login 页面校验 bcrypt 加密的密码,添加 cookie 和 logout 页面删除 cookie 的过程很简单,就不说啥了。要点在于这个授权部分。这是 RBAC(基于角色的访问控制)系统,所以我这里特意演示了一个相对复杂的定义:</p>
<ol>
<li>john 用户定义了自己的 group 为 user。</li>
<li>定义首页的授权目标(<code class="highlighter-rouge">target</code>)为:group 为 user <strong>或者</strong> admin 的用户。注意这里的写法是 <code class="highlighter-rouge">[{xxx},{yyy}]</code>。如果写法是 <code class="highlighter-rouge">[{xxx, yyy}]</code>,那含义就不是<strong>或者</strong>而是<strong>并且</strong>了。</li>
<li><code class="highlighter-rouge">target</code> 里可以用以下对象:<code class="highlighter-rouge">credentials</code>, <code class="highlighter-rouge">connection</code>, <code class="highlighter-rouge">query</code>, <code class="highlighter-rouge">param</code>, <code class="highlighter-rouge">request</code>。注意这里引用 key 的写法是冒号(比如从 HTTP header 中获取主机名的写法为 <code class="highlighter-rouge">connection:host</code>)。</li>
<li>定义该目标的授权方式为 <code class="highlighter-rouge">apply</code>,即还需要后续判断。如果直接就授权,那应该写作 <code class="highlighter-rouge">effect</code>。</li>
<li><code class="highlighter-rouge">apply</code> 方式定义为 <code class="highlighter-rouge">permit-overrides</code>。意即:后续条件只要满足一个就允许,否则拒绝。<code class="highlighter-rouge">deny-overrides</code> 反之亦然。</li>
<li>开始定义具体的 <code class="highlighter-rouge">policies</code> 集合。同样格式也是或的关系。这里如果没有复杂需求也可以直接开始 <code class="highlighter-rouge">rules</code> 定义。</li>
<li>每个小 policy 里也是一个完整的授权定义,也有自己的 <code class="highlighter-rouge">target</code> 等。</li>
<li>开始 <code class="highlighter-rouge">rules</code> 定义。<code class="highlighter-rouge">rules</code> 里的条件相当于是 if-else 关系。</li>
</ol>
<p>最终本文示例的意思就是:</p>
<p><strong>首页只允许 admin 组全体用户加上 user 组里的 john 用户访问。</strong></p>
<hr />
<p>简单的 hello world 示意如此。再往深了走,可以把 user 定义、policy 定义都搬到数据库里。再再往深里走。可以把 Kibana 里所有的 route 都用这块做一个接管。就大功告成了。</p>
<p>不过在 hapi.js 上动手,只是对后端接口做了授权控制,前端页面看起来还是都一样的。如果为了美观,就可以配合加上 <a href="https://github.com/plandem/angular-rbac">angular-rbac</a>,对前端页面也稍作修改,针对不同 user 展示不同内容。</p>
Lucene 查询中的距离查询(proximity query)
2016-04-04T00:00:00+08:00
elasticsearch
Lucene
http://chenlinux.com/2016/04/04/lucene-proximity-querystring
<p>我们在使用 ELK 的时候,使用 Lucene querystring 语法的机会,远超过使用 Elasticsearch 的 query DSL。毕竟在搜索框里写语法比自己拼 JSON 简单多了。</p>
<p>不过一般我们用的 querystring 语法总是最简单的几样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>text
key:value
key:"term"
k1:v1 AND NOT k2:v2
</code></pre>
</div>
<p>80% 的情况下,这几个用法也就足够了。但总有剩下的 20% 的情况,还是需要我们来了解一些更复杂的语法。</p>
<p>举一个还算通用的场景:<strong>我们在 ELK 里索引了访问日志。这时候需要查一下以 <code class="highlighter-rouge">/api/login</code> 开头的 URL 们的情况。</strong></p>
<p>我们没法确定 URL 里是不是只有 <code class="highlighter-rouge">/api/login</code> 一种可能。没准可能还有 <code class="highlighter-rouge">/api/oauth/login</code> 呢?没准可能还有 <code class="highlighter-rouge">login/weibo/api</code> 呢?</p>
<p>一般来说,日志进 ELK 都是采用标准分词器的。而很巧,<code class="highlighter-rouge">/</code> 就是标准分词器的停止词之一。所以,我们在搜索框里写 <code class="highlighter-rouge">api/login</code> 等效于 <code class="highlighter-rouge">api login</code>。那么太多可能都可以命中了。</p>
<p>这个时候,Lucene 查询语法里的距离查询(proximity query)就可以帮忙了:</p>
<p><code class="highlighter-rouge">url:"api login"</code></p>
<p>看起来很简单,无非是给加了一对双引号?!</p>
<p>没错,加引号以后,意味着这个短语查询必须是有序的,即只能命中<em>先出现api,再出现login</em>的文本了。这下就把 <code class="highlighter-rouge">login/weibo/api</code> 排除掉了。</p>
<p>其次,Lucene 距离查询默认的距离为 0,即只能命中<em>出现api之后,下一个term必须为login</em>的文本了。这些就把 <code class="highlighter-rouge">/api/oauth/login</code> 也排除了。</p>
<p>当然,如果这时候你日志里除了 <code class="highlighter-rouge">api/login</code> 还有 <code class="highlighter-rouge">api,login</code> 之类的文本,也是会命中的。不过在 url 字段里出现这个的概率不大,可以无视了~</p>
<p>如果你要搜的就是 <code class="highlighter-rouge">/api/oauth/login</code>,但是你不记得中间这个是不是 oauth,也可能是其他的吧,怎么办?</p>
<p><code class="highlighter-rouge">url:"api login"~1</code></p>
<p>后面加波浪线和距离即可。</p>
用火焰图看 elasticsearch 的资源占用
2016-04-01T00:00:00+08:00
monitor
nodejs
elasticsearch
flamegraph
http://chenlinux.com/2016/04/01/javaflamegraph
<p>我们都很习惯在压测 nginx 等服务的时候,利用 systemtap 完成 flamegraph 火焰图来看具体哪个函数占用 CPU 资源过多了。那么,对 Java 实现的 elasticsearch,有没有类似办法呢?</p>
<p>JDK 自带有 jstack 命令,可以获取相关信息,其实只要一个可视化的过程就行了。而社区也有人早已做好。下面就是 nodejs 的 javaflamegraph 库的安装使用过程:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget https://nodejs.org/download/release/v5.10.0/node-v5.10.0-linux-x64.tar.gz --no-check-certificate
tar zxvf node-v5.10.0-linux-x64.tar.gz
cd node-v5.10.0-linux-x64
./bin/npm install javaflamegraph
../../bin/npm run start `ps aux|grep elasticsearc[h]|awk '{print $2}'`
</code></pre>
</div>
<p>确保 jstack 命令可用(flame-gen.sh 里是直接调用的,注意 PATH),确保当前目录可写。</p>
<p>等待几十秒后,中止进程。用浏览器打开当前目录下的 flame.html。可以看到如下效果:</p>
<p><img src="/images/uploads/esflame0.png" alt="" /></p>
<p>鼠标大概在上面移动一下,可以看到大概 segment merge 和 bulk thread 各占了 ES 进程资源消耗的半壁江山。</p>
<p>我们再点击一个 bulk thread,看看细节:</p>
<p><img src="/images/uploads/esflame1.png" alt="" /></p>
<p>可以看到其中 primary 和 replica 各占一部分。两者各自包括各自的三块:lucene 的 indexWriter、loadCurrentVersionFromIndex、translog。</p>
<p>在集群压测中,这三块的占比大概是后面两个加起来不到15%的样子,如果做日志场景,其实有可能用不上 version 检查,可以省掉大概 10% 的资源消耗。不过,谁都难免有异常要 retry,通过 version 检查避免重复的 indexing,也是有利的。所以总体来说:elasticsearch 在索引性能方面,做的应该是挺好了。要提高这个速度,可能更需要关心的是 lucene 层面的方案,比如分词方式、结构化程度等等~</p>
机器战胜人类了,伺候机器的运维呢?
2016-03-19T00:00:00+08:00
monitor
http://chenlinux.com/2016/03/19/machine-vs-ops
<p>2016 年 3 月最火爆的新闻,莫过于谷歌的 alphago 机器 4:1 大胜李世乭了。一时间各界议论纷纷,我的前同事,运维界非著名段子手 <a href="http://weibo.com/30007147">@orroz</a> 在自己微博上写了两段话:</p>
<blockquote>
<p>『跟其他运维工程师觉得这个职业将消失不同。我是对运维职业是持极端乐观态度的,也许运维职业将是人类最后一个职业。很可能祂们在能自理之前还需要我们伺候。。。也说不定,某几个运维工程师因为某种不知道的原因还会被祂们当宠物留下来,成为人类的最后的延续。』<br />
<img src="http://ww4.sinaimg.cn/large/6673053fgw1f1qv2q6duaj209h0e83z4.jpg" alt="" /><br />
『我终于明白这个图片的寓意了,它其实预示了人类的未来命运。』</p>
</blockquote>
<p>看完一笑~</p>
<p>但是笑完以后,回头想想,运维和围棋手,其实还真是有相像的地方:传统说法中,与研发相比,运维总被认为是『更靠经验的』;一如我们说『人类棋手的经验和大局观』。</p>
<p>我们知道,运维的『操作』,已经是可替代的了,IaaS、PaaS、运维自动化,诸多概念的落地,环境部署、软件安装不再是运维的主要工作职责。运维的职位名称,从系统管理员到运维工程师到产品工程师到站点可靠性工程师,一步步远离了基础设备层面。</p>
<p>那,有没有可能,运维的『经验』,也是可以被机器替代掉的呢?</p>
<h3 id="section">运维经验</h3>
<p>我们先看看运维的经验到底是什么?</p>
<ul>
<li>一个 4 核 CPU 的服务器,loadavg 跑到 10+,我们就会说:负载过高了。应对办法最简单的就是『加机器』。</li>
<li>一个 web 服务,每秒请求超过 1000,响应变慢了,我们就会说:还在用 apache 啊,快换 nginx 吧。</li>
<li>要是动态服务呢,就会说:做个动静分离呗,加个缓存层呗。</li>
</ul>
<p>这就是运维届的『定式』和『俗手』。</p>
<p>但是不巧,定式并不能一路保送我们最后顺利完工。</p>
<p>就好像这五场世纪大战一开始,人类棋手总觉得 alphago 水平不行——『职业初段的人都应该知道下这里才对啊』。但是一百多手不知不觉过去,局面就是不利了!</p>
<h3 id="section-1">经验的坑</h3>
<p>比方前面说的第一条经验,这几乎已经是运维共识了。但是把环境考虑进来:这如果是一台虚拟机呢?这如果挂载的是一个远端存储呢?这如果运行的是一个无法水平扩展的事务系统呢?</p>
<p>是的,『加机器』只能死的更惨。(此处应配有那两把著名的刘强东之刀)</p>
<p>所以,经验是否真的能成立,有赖于更复杂和深层次的分析。就像围棋依赖于算力一样。</p>
<h3 id="section-2">大数据那么美好么</h3>
<p>文章写到这里,似乎我要开始鼓吹运维届要如何如何上马大数据乃至机器学习了?</p>
<p>这种玩法看起来确实高大上,但实际上,并没有那么美好!我们不要忘了:运维始终是一个 IT 支出向的工作。DevOps 运动中说运维加快部署就是赚钱,那也是间接的。花钱是直接的。还是引用另一个微博上有关 alphago 的段子:</p>
<blockquote>
<p>alphago 跑了 1000 个 CPU,李世乭吃了一餐饭,比一下资源消耗就知道谁赢了。</p>
</blockquote>
<p>运维工程师拥有前所未有的多的机器数据,理论上当然可以通过大数据挖掘,通过机器学习获得相当多的收获。但是这些收获跟能间接带来的收益相比,性价比如何呢?</p>
<p>拿监控数据来说,我们知道监控产生的,大多是时序数值。对于时序数值的分析,金融界早有数十年的算法研究和积累。运维工程师照搬过来,未尝不可。但这其中一些算法消耗的 CPU 运算,没准比本身业务系统运行消耗的还高,那这个花费显然就不可能投入。</p>
<p>《人工智能的未来》作者,神经学家 Jeff Hawkins 成立的 numenta 公司曾经对市面上各种号称处理时序数据异常探测或者预测分析的开源实现做了对比性测试。结果,真正能满足『时序、动态』前提的都不多,有些算法长达一个小时都完成不了测试。更好玩的是:有的测试场景中,随机选异常点都有 25.9% 的准确率。</p>
<p>测试见:<a href="https://github.com/numenta/NAB">https://github.com/numenta/NAB</a>。(当然我这里不是来推销说 HTM 算法是人工智能未来,毕竟 alphago 是 DNN 呢)</p>
<h3 id="section-3">废话这么多,到底怎么办</h3>
<p>又要深入分析,又要控制能耗。最好的办法,就是把不确定性降低,在一个较完善的运维体系框架基础上做数据分析,可以大大缩小数据集,降低复杂度。</p>
<p>运维体系怎么才算完善,已经有很多文章在讲了。以数据分析为目的的话,个人推荐王津银的数据驱动运维系列文章。</p>
<p>分析本身如何入手,其实简单算法也未必不好。百度云在 SREcon15 上的分享,同样推荐观看。在线数据通过简单的 3-sigma、ks-test、holt-winters、LOESS 来生成异常点,然后仅对异常点采用 Viterbi 计算同比的异常区域发出实际告警,配合通用的 tracing 调用链系统使用。</p>
<p>最后回到文章开头的段子:机器为啥留下几个运维工程师?或许因为这几个运维当初给机器安排的都是算 3-sigma 这样轻松的活,一报还一报吧:)</p>
juttle 可视化界面介绍
2016-03-16T00:00:00+08:00
logstash
elasticsearch
javascript
nodejs
http://chenlinux.com/2016/03/16/juttle-viz-intro
<p>上篇介绍了一下怎么用 juttle 交互式命令行查看表格式输出。juttle 事实上还提供了一个 web 服务器,做数据可视化效果,这个同样是用 juttle 语言描述配置。</p>
<p>我们已经在上一篇安装好了 <code class="highlighter-rouge">juttle-engine</code> 模块,那么直接启动服务器即可:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>~$ juttle-engine -d
</code></pre>
</div>
<p>然后浏览器打开 <code class="highlighter-rouge">http://localhost:8080</code> 就能看到页面了。注意,请使用 Chrome v45 以上版本或者 Safari 等其他浏览器,否则有个 Array 上的 bug。</p>
<p>但是目前这个页面上本身不提供输入框直接写 juttle 语言。所以需要我们把 juttle 语言写成脚本文件,再来通过页面加载。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>~$ cat > ~/test.juttle <<EOF
read elastic -index 'logstash-*' -from :-2d: -to :now: 'MacBook-Pro'
| reduce -every :1h: count() by 'path.raw'
| (
view timechart -row 0 -col 0;;
view table -height 200 -row 1 -col 0;
view piechart -row 1 -col 0;
);
(
read elastic -index 'logstash-*' -from :-2d: -to :-1d: 'MacBook-Pro' AND '/var/log/system.log'
| reduce -every :1h: count();
read elastic -index 'logstash-*' -from :-1d: -to :now: 'MacBook-Pro' AND '/var/log/system.log'
| reduce -every :1h: count();
)
| (
view timechart -duration :1 day: -overlayTime true -height 400 -row 0 -col 1 -title 'syslog hour-on-hour';
view table -height 200 -row 1 -col 1;
);
EOF
</code></pre>
</div>
<p>然后访问 <code class="highlighter-rouge">http://localhost:8080?path=/test.juttle</code>,注意这里的path参数的写法,这个/其实指的是你运行 <code class="highlighter-rouge">juttle-engine</code> 命令的时候的路径,而不是真的设备根目录。</p>
<p>就可以在浏览器上看到如下效果:</p>
<p><img src="/images/uploads/juttle-viz.png" alt="" /></p>
<p>页面上还有一行有关 <code class="highlighter-rouge">path.raw</code> 的 WARNING 提示,那是因为 juttle 目前对 elasticsearch 的 mapping 解析支持的不是很好,但是不影响使用,可以不用管。</p>
<h2 id="section">可视化相关指令介绍</h2>
<p>我们可以看到这次的 juttle 脚本,跟昨天在命令行下运行的几个区别:</p>
<ol>
<li>我们用上了 <code class="highlighter-rouge">()</code>,这是 juttle 的一大特技,对同一结果并联多个 view ,或者并联多个输入结果做相同的后续处理等等。</li>
<li>我们对 view 用上了 <code class="highlighter-rouge">row</code> 和 <code class="highlighter-rouge">col</code> 参数,用来指定他们在页面上的布局。</li>
<li>有一个 <code class="highlighter-rouge">timechart</code> 我们用了 <code class="highlighter-rouge">-durat :1d: -overlayTime true</code> 参数。这是 <code class="highlighter-rouge">timechart</code> 独有的参数,专门用来实现同比环比的。在图上的效果大家也可以看到了。不过目前也有小问题,就是鼠标放到图上的时候,只能看到第二个结果的指标说明,看不到第一个的。</li>
</ol>
juttle 介绍
2016-03-16T00:00:00+08:00
logstash
elasticsearch
javascript
nodejs
http://chenlinux.com/2016/03/16/juttle-intro
<p>juttle 是一个 nodejs 项目,专注于数据处理和可视化。它自定义了一套自己的 DSL,提供交互式命令行、程序运行、界面访问三种运行方式。</p>
<p>在 juttle 的 DSL 中,可以用 <code class="highlighter-rouge">|</code> 管道符串联下列指令实现数据处理:</p>
<ul>
<li>通过 read 指令读取来自 http、file、elasticsearch、graphite、influxdb、opentsdb、mysql 等数据源,</li>
<li>通过 filter 指令及自定义的 JavaScript 函数做数据过滤,</li>
<li>通过 reduce 指令做数据聚合,</li>
<li>通过 join 指令做数据关联,</li>
<li>通过 write 指令做数据转储,</li>
<li>通过 view 指令做数据可视化。</li>
</ul>
<p>更关键的,可以用 <code class="highlighter-rouge">()</code> 并联同一层级的多条指令进行处理。</p>
<p>看起来非常有意思的项目,赶紧试试吧。</p>
<h2 id="section">安装部署</h2>
<p>既然说了这是一个 nodejs 项目,自然是通过 npm 安装了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sudo npm install -g juttle
sudo npm install -g juttle-engine
</code></pre>
</div>
<p>注意,如果是在 MacBook 上安装的话,一定要先通过 AppStore 安装好 Xcode 并确认完 license。npm 安装依赖的 sqlite3 的时候没有 xcode 会僵死在那。</p>
<p>juttle 包提供了命令行交互,juttle-engine 包提供了网页访问的服务器。</p>
<p>juttle 的配置文件默认读取位置是 <code class="highlighter-rouge">$HOME/.juttle/config.json</code>。比如读取本机 elasticsearch 的数据,那么定义如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nt">"adapters"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"elastic"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"localhost"</span><span class="p">,</span><span class="w">
</span><span class="nt">"port"</span><span class="p">:</span><span class="w"> </span><span class="mi">9200</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
</div>
<p>甚至可以读取多个不同来源的 elasticsearch,这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nt">"adapters"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"elastic"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"one"</span><span class="p">,</span><span class="w">
</span><span class="nt">"address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"localhost"</span><span class="p">,</span><span class="w">
</span><span class="nt">"port"</span><span class="p">:</span><span class="w"> </span><span class="mi">9200</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"two"</span><span class="p">,</span><span class="w">
</span><span class="nt">"address"</span><span class="p">:</span><span class="w"> </span><span class="s2">"localhost"</span><span class="p">,</span><span class="w">
</span><span class="nt">"port"</span><span class="p">:</span><span class="w"> </span><span class="mi">9201</span><span class="w">
</span><span class="p">}],</span><span class="w">
</span><span class="nt">"influx"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"http://examples_influxdb_1:8086"</span><span class="p">,</span><span class="w">
</span><span class="nt">"user"</span><span class="p">:</span><span class="w"> </span><span class="s2">"root"</span><span class="p">,</span><span class="w">
</span><span class="nt">"password"</span><span class="p">:</span><span class="w"> </span><span class="s2">"root"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
</div>
<h2 id="section-1">命令行运行示例</h2>
<p>配置完成,就可以交互式命令行运行了。终端输入 <code class="highlighter-rouge">juttle</code> 回车进入交互界面。我们输入下面一段查询:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>juttle> read elastic -id one -index 'logstash-*' -from :1 year ago: -to :now: 'MacBook-Pro' | reduce -every :1h: c = count() by path | filter c > 1000 | put line = 10000 | view table -columnOrder 'time', 'c', 'line', 'path'
</code></pre>
</div>
<p>输出如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>┌────────────────────────────────────┬──────────┬──────────┬─────────────────────────────┐
│ time │ c │ line │ path │
├────────────────────────────────────┼──────────┼──────────┼─────────────────────────────┤
│ 2016-03-02T10:00:00.000Z │ 4392 │ 10000 │ /var/log/system.log │
├────────────────────────────────────┼──────────┼──────────┼─────────────────────────────┤
│ 2016-03-02T11:00:00.000Z │ 4818 │ 10000 │ /var/log/system.log │
├────────────────────────────────────┼──────────┼──────────┼─────────────────────────────┤
│ 2016-03-02T12:00:00.000Z │ 2038 │ 10000 │ /var/log/system.log │
├────────────────────────────────────┼──────────┼──────────┼─────────────────────────────┤
│ 2016-03-02T13:00:00.000Z │ 1826 │ 10000 │ /var/log/system.log │
├────────────────────────────────────┼──────────┼──────────┼─────────────────────────────┤
│ 2016-03-02T15:00:00.000Z │ 10267 │ 10000 │ /var/log/system.log │
├────────────────────────────────────┼──────────┼──────────┼─────────────────────────────┤
│ 2016-03-02T16:00:00.000Z │ 10999 │ 10000 │ /var/log/system.log │
├────────────────────────────────────┼──────────┼──────────┼─────────────────────────────┤
│ 2016-03-02T17:00:00.000Z │ 3528 │ 10000 │ /var/log/system.log │
├────────────────────────────────────┼──────────┼──────────┼─────────────────────────────┤
│ 2016-03-03T00:00:00.000Z │ 2498 │ 10000 │ /var/log/system.log │
├────────────────────────────────────┼──────────┼──────────┼─────────────────────────────┤
│ 2016-03-03T03:00:00.000Z │ 4600 │ 10000 │ /var/log/system.log │
├────────────────────────────────────┼──────────┼──────────┼─────────────────────────────┤
│ 2016-03-03T04:00:00.000Z │ 7751 │ 10000 │ /var/log/system.log │
├────────────────────────────────────┼──────────┼──────────┼─────────────────────────────┤
│ 2016-03-03T05:00:00.000Z │ 3249 │ 10000 │ /var/log/system.log │
├────────────────────────────────────┼──────────┼──────────┼─────────────────────────────┤
│ 2016-03-03T06:00:00.000Z │ 5715 │ 10000 │ /var/log/system.log │
├────────────────────────────────────┼──────────┼──────────┼─────────────────────────────┤
│ 2016-03-03T07:00:00.000Z │ 4374 │ 10000 │ /var/log/system.log │
├────────────────────────────────────┼──────────┼──────────┼─────────────────────────────┤
│ 2016-03-03T08:00:00.000Z │ 2600 │ 10000 │ /var/log/system.log │
└────────────────────────────────────┴──────────┴──────────┴─────────────────────────────┘
</code></pre>
</div>
<p>漂亮的终端表格!</p>
<h2 id="section-2">警告</h2>
<p>需要注意的是,juttle 和 es-hadoop 一样,也是通过 RESTful API 和 elasticsearch 交互,所以除了个别已经提前实现好了的 reduce 方法可以转换成 aggregation 以外,其他的 juttle 指令,都是通过 query 把数据拿回来以后,由 juttle 本身做的运算处理。juttle-adapter-elastic 模块的 <code class="highlighter-rouge">DEFAULT_FETCH_SIZE</code> 设置是 10000 条。</p>
<p>而比 es-hadoop 更差的是,因为 juttle 是单机程序,它还没有像 es-hadoop 那样并发 partition 直连每个 elasticsearch 的 shard 做并发请求。</p>
Kibana4 服务器端插件开发
2016-01-27T00:00:00+08:00
logstash
kibana
elasticsearcch
javascript
node.js
watcher
http://chenlinux.com/2016/01/27/kibana-server-plugin-develop
<p>我在 ELK Stack 中文指南的 <a href="http://kibana.logstash.es/content/kibana/v4/source-code-analysis/visualize_app.html">visualize 解析</a>一节介绍了如何给 Kibana4 开发浏览器端的可视化插件。Kibana4 跟 Kibana3 比,最大的一个变化是有了独立的 node.js 服务器端。那么同样的,也就有了服务器端的 Kibana4 插件。最明显的一个场景:我们可以在 node.js 里跑定时器做 Elasticsearch 的告警逻辑了!</p>
<p>本文示例一个最基础的 Kibana4 告警插件开发。只演示基础的定时器和 Kibana4 插件规范,实际运用中,肯定还涉及历史记录,告警项配置更新等。请读者不要直接 copy-paste。</p>
<p>首先,我们尽量沿袭 Elastic 官方的 watcher 产品的告警配置设计。也新建一个索引,里面是具体的配置内容:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="err">#</span><span class="w"> </span><span class="err">curl</span><span class="w"> </span><span class="err">-XPUT</span><span class="w"> </span><span class="err">http://</span><span class="mf">127.0</span><span class="err">.</span><span class="mf">0.1</span><span class="err">:</span><span class="mi">9200</span><span class="err">/watcher/watch/error_status</span><span class="w"> </span><span class="err">-d'</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nt">"trigger"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"schedule"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"interval"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"60"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"input"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"search"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"request"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"indices"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"<logstash-{now/d}>"</span><span class="p">,</span><span class="w"> </span><span class="s2">"<logstash-{now/d-1d}>"</span><span class="w"> </span><span class="p">],</span><span class="w">
</span><span class="nt">"body"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"query"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"filtered"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"query"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"match"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"host"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"MacBook-Pro"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="nt">"filter"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"range"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"@timestamp"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"from"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"now-5m"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"condition"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"script"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"script"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"payload.hits.total > 0"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"transform"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"search"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"request"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"indices"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"<logstash-{now/d}>"</span><span class="p">,</span><span class="w"> </span><span class="s2">"<logstash-{now/d-1d}>"</span><span class="w"> </span><span class="p">],</span><span class="w">
</span><span class="nt">"body"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"query"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"filtered"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"query"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"match"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"host"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"MacBook-Pro"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="nt">"filter"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"range"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"@timestamp"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"from"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"now-5m"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"aggs"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"topn"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"terms"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"field"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"path.raw"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"actions"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"email_admin"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"throttle_period"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"15m"</span><span class="p">,</span><span class="w">
</span><span class="nt">"email"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"to"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"admin@domain"</span><span class="p">,</span><span class="w">
</span><span class="nt">"subject"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Found Error Events"</span><span class="p">,</span><span class="w">
</span><span class="nt">"priority"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"high"</span><span class="p">,</span><span class="w">
</span><span class="nt">"body"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Top10 paths:\n\t \n"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="err">'</span><span class="w">
</span></code></pre>
</div>
<p>我们可以看到,跟原版的相比,只改动了很小的一些地方:</p>
<ol>
<li>为了简便,<code class="highlighter-rouge">interval</code> 固定写数值,没带 <code class="highlighter-rouge">s/m/d/H</code> 之类的单位;</li>
<li><code class="highlighter-rouge">condition</code> 里直接使用了 JavaScript,这点也是 ES 2.x 的 mapping 要求跟 watcher 本身有冲突的一个地方:watcher的 <code class="highlighter-rouge">"ctx.payload.hits.total" : { "gt" : 0 }</code> 这种写法,如果是普通索引,会因为字段名里带 <code class="highlighter-rouge">.</code> 直接写入失败的;</li>
<li>因为是在 Kibana 里面运行,所以从 ES 拿到的只有 payload(也就是查询响应),所以把里面的 <code class="highlighter-rouge">ctx.</code> 都删掉了。</li>
</ol>
<p>好,然后创建插件:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>cd kibana-4.3.0-darwin-x64/src/plugins
mkdir alert
</code></pre>
</div>
<p>在自定义插件目录底下创建 <code class="highlighter-rouge">package.json</code> 描述:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"alert"</span><span class="p">,</span><span class="w">
</span><span class="nt">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0.0.1"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
</div>
<p>以及最终的 <code class="highlighter-rouge">index.js</code> 代码:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="s1">'use strict'</span><span class="p">;</span>
<span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">kibana</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">later</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'later'</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">_</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'lodash'</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">mustache</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'mustache'</span><span class="p">);</span>
<span class="k">return</span> <span class="k">new</span> <span class="nx">kibana</span><span class="p">.</span><span class="nx">Plugin</span><span class="p">({</span>
<span class="na">init</span><span class="p">:</span> <span class="kd">function</span> <span class="nx">init</span><span class="p">(</span><span class="nx">server</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">client</span> <span class="o">=</span> <span class="nx">server</span><span class="p">.</span><span class="nx">plugins</span><span class="p">.</span><span class="nx">elasticsearch</span><span class="p">.</span><span class="nx">client</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">sched</span> <span class="o">=</span> <span class="nx">later</span><span class="p">.</span><span class="nx">parse</span><span class="p">.</span><span class="nx">text</span><span class="p">(</span><span class="s1">'every 10 minute'</span><span class="p">);</span>
<span class="nx">later</span><span class="p">.</span><span class="nx">setInterval</span><span class="p">(</span><span class="nx">doalert</span><span class="p">,</span> <span class="nx">sched</span><span class="p">);</span>
<span class="kd">function</span> <span class="nx">doalert</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">getCount</span><span class="p">().</span><span class="nx">then</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">resp</span><span class="p">){</span>
<span class="nx">getWatcher</span><span class="p">(</span><span class="nx">resp</span><span class="p">.</span><span class="nx">count</span><span class="p">).</span><span class="nx">then</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">resp</span><span class="p">){</span>
<span class="nx">_</span><span class="p">.</span><span class="nx">each</span><span class="p">(</span><span class="nx">resp</span><span class="p">.</span><span class="nx">hits</span><span class="p">.</span><span class="nx">hits</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">hit</span><span class="p">){</span>
<span class="kd">var</span> <span class="nx">watch</span> <span class="o">=</span> <span class="nx">hit</span><span class="p">.</span><span class="nx">_source</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">every</span> <span class="o">=</span> <span class="nx">watch</span><span class="p">.</span><span class="nx">trigger</span><span class="p">.</span><span class="nx">schedule</span><span class="p">.</span><span class="nx">interval</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">watchSched</span> <span class="o">=</span> <span class="nx">later</span><span class="p">.</span><span class="nx">parse</span><span class="p">.</span><span class="nx">recur</span><span class="p">().</span><span class="nx">every</span><span class="p">(</span><span class="nx">every</span><span class="p">).</span><span class="nx">second</span><span class="p">();</span>
<span class="kd">var</span> <span class="nx">wt</span> <span class="o">=</span> <span class="nx">later</span><span class="p">.</span><span class="nx">setInterval</span><span class="p">(</span><span class="nx">watching</span><span class="p">,</span> <span class="nx">watchSched</span><span class="p">);</span>
<span class="kd">function</span> <span class="nx">watching</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">request</span> <span class="o">=</span> <span class="nx">watch</span><span class="p">.</span><span class="nx">input</span><span class="p">.</span><span class="nx">search</span><span class="p">.</span><span class="nx">request</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">condition</span> <span class="o">=</span> <span class="nx">watch</span><span class="p">.</span><span class="nx">condition</span><span class="p">.</span><span class="nx">script</span><span class="p">.</span><span class="nx">script</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">transform</span> <span class="o">=</span> <span class="nx">watch</span><span class="p">.</span><span class="nx">transform</span><span class="p">.</span><span class="nx">search</span><span class="p">.</span><span class="nx">request</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">actions</span> <span class="o">=</span> <span class="nx">watch</span><span class="p">.</span><span class="nx">actions</span><span class="p">;</span>
<span class="nx">client</span><span class="p">.</span><span class="nx">search</span><span class="p">(</span><span class="nx">request</span><span class="p">).</span><span class="nx">then</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">payload</span><span class="p">){</span>
<span class="kd">var</span> <span class="nx">ret</span> <span class="o">=</span> <span class="nb">eval</span><span class="p">(</span><span class="nx">condition</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">ret</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">client</span><span class="p">.</span><span class="nx">search</span><span class="p">(</span><span class="nx">transform</span><span class="p">).</span><span class="nx">then</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">payload</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">_</span><span class="p">.</span><span class="nx">each</span><span class="p">(</span><span class="nx">_</span><span class="p">.</span><span class="nx">values</span><span class="p">(</span><span class="nx">actions</span><span class="p">),</span> <span class="kd">function</span><span class="p">(</span><span class="nx">action</span><span class="p">){</span>
<span class="k">if</span><span class="p">(</span><span class="nx">_</span><span class="p">.</span><span class="nx">has</span><span class="p">(</span><span class="nx">action</span><span class="p">,</span> <span class="s1">'email'</span><span class="p">))</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">subject</span> <span class="o">=</span> <span class="nx">mustache</span><span class="p">.</span><span class="nx">render</span><span class="p">(</span><span class="nx">action</span><span class="p">.</span><span class="nx">email</span><span class="p">.</span><span class="nx">subject</span><span class="p">,</span> <span class="p">{</span><span class="s2">"payload"</span><span class="p">:</span><span class="nx">payload</span><span class="p">});</span>
<span class="kd">var</span> <span class="nx">body</span> <span class="o">=</span> <span class="nx">mustache</span><span class="p">.</span><span class="nx">render</span><span class="p">(</span><span class="nx">action</span><span class="p">.</span><span class="nx">email</span><span class="p">.</span><span class="nx">body</span><span class="p">,</span> <span class="p">{</span><span class="s2">"payload"</span><span class="p">:</span><span class="nx">payload</span><span class="p">});</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">subject</span><span class="p">,</span> <span class="nx">body</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="p">});</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">getCount</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">client</span><span class="p">.</span><span class="nx">count</span><span class="p">({</span>
<span class="na">index</span><span class="p">:</span><span class="s1">'watcher'</span><span class="p">,</span>
<span class="na">type</span><span class="p">:</span><span class="s2">"watch"</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">getWatcher</span><span class="p">(</span><span class="nx">count</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">client</span><span class="p">.</span><span class="nx">search</span><span class="p">({</span>
<span class="na">index</span><span class="p">:</span><span class="s1">'watcher'</span><span class="p">,</span>
<span class="na">type</span><span class="p">:</span><span class="s2">"watch"</span><span class="p">,</span>
<span class="na">size</span><span class="p">:</span><span class="nx">count</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="p">};</span>
</code></pre>
</div>
<p>其中用到了两个 npm 模块,later 模块用来实现定时器和 crontab 文本解析,mustache 模块用来渲染邮件内容模板,这也是 watcher 本身采用的渲染模块。</p>
<p>需要安装一下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>npm install later
npm install mustache
</code></pre>
</div>
<p>然后运行 <code class="highlighter-rouge">./bin/kibana</code>,就可以看到终端上除了原有的内容以外,还会定期输出 alert 的 email 内容了。</p>
<h2 id="section">要点解释</h2>
<p>这个极简示例中,主要有两段:</p>
<ol>
<li>注册为插件</li>
</ol>
<div class="highlighter-rouge"><pre class="highlight"><code>module.exports = function (kibana) {
return new kibana.Plugin({
init: function init(server) {
</code></pre>
</div>
<p>如果是浏览器端插件,这块应该是:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>module.exports = function (kibana) {
return new kibana.Plugin({
uiExports: {
</code></pre>
</div>
<ol>
<li>引用 ES client</li>
</ol>
<div class="highlighter-rouge"><pre class="highlight"><code> init: function init(server) {
var client = server.plugins.elasticsearch.client;
</code></pre>
</div>
<p>这里通过调用 <code class="highlighter-rouge">server.plugins</code> 来直接引用 Kibana 里其他插件里的对象。这样,alert 插件就可以跟其他功能共用同一个 ES client,免去单独配置自己的 ES 设置项和新开网络连接的资源消耗。</p>
2015 年度个人总结
2015-12-27T00:00:00+08:00
http://chenlinux.com/2015/12/27/report-of-this-year
<p>又一年过去了。2015 年在博客上发表文章的时间大幅度减少,全年只写了 23 篇博客,其实还有一小半是翻译。但是个人总结还是要写的,写在博客上,因为别的平台肯定不适合发这个……</p>
<h2 id="section">写作</h2>
<p>整个 2015 年,所有的时间都用在了日志分析领域。博客上 23 篇都是 ELK、Rsyslog、Spark 相关话题。而且把去年动手写的 gitbook 正式整理完善,在机械工业出版社出版了。取名依然是个人弱项,最后咬咬牙,就叫《ELK Stack权威指南》吧。</p>
<p>写书是件严肃的事情,换一家出版社,体验更深。在书稿完成,即将上架的时候,因为书名上这几个英文字母大小写、带不带空格的问题,编辑和我花了两天时间,收集各式资料,意图确认一个最官方最权威的拼写用法。</p>
<p>其实 Elastic.co 官方可能并不是特别在意这个问题。因为最终我在官网上发现了两种写法,在官推上,发现了四种写法……甚至最近的官方文档,又有直接叫 Elastic Stack 的,估计是准备强化公司品牌。不过在亚马逊上,看见一个新书预告,一位欧洲同好预备明年 2 月出版一本《ELK Stack Cookbook》。所以,最后我决定都统一为”ELK Stack”。</p>
<p>为了赶上 ESCC 北京站之前出版,书在十月底上架,不太巧的立马就碰上京东十一月两次搞活动,新书总是会被缺货的,于是有热心的朋友来问我,就只好都劝他们去互动出版网上购买。结果十一月这本书赫然变成互动出版网上的分类第一名。到前两天,看京东的 2015 年度计算机类书籍榜单,这本书在潜力榜排第 10 名。总之,应该还是对得起朋友,对得起读者,对得起自己了。</p>
<p>然而和一年前的总结一样,一年后我依然犯了一个错误:新书还是忘了在致谢中感谢老婆大人……事实上每个我在埋头写书的夜晚,她都一个人坐在离我两米外的地方直等到慢慢睡着。苍天,看来 2016 年我还得在写一本弥补这个遗憾……</p>
<p>和去年张罗 PerlChina Advent 一样,今年又尝试搞了一次 <a href="http://elasticsearch.cn/topic/advent">ELK Stack Advent</a>。最后个人写了 16 篇,比去年 11 篇略微上升。真的只能说:要一个人干这种事情,太难太难了。感谢 medcl 的支持,感谢 mdecl、wood、childe 几位一块完成这次 advent 的伙伴。ELK 社区跟 Perl 社区可不一样,Perl 是有这项传统的,只要有人起个头,找齐人写还算容易。而 ELK 作为新社区,又刚刚举办过大会讲过各种分享了,真是感谢伙伴们榨出来的干货。日本 qiita 社区上登记了快 500 个 advent,最后坚持写完 24 篇的社区,也是少之又少。毅力和言必信行必果,绝对是一种宝贵的财富。</p>
<h2 id="section-1">演讲</h2>
<p>2015 年从一开始,就在到处参会演讲和交流分享。一直到这个月才中断。算上 2014 年总结中说的 4 个月,也就是我连续 15 个月在大小聚会中宣讲了 ELK Stack 和周边。最开始是一次网络课堂,讲没几分钟,有人在侧边框的聊天栏发『老师内容讲挺好的,别着急,声音别发抖』……当时准备的 PPT 不到 40 页。一路经过 WOT、infoQ、中华数据库大会、运维帮,到最后 PHPConf 的时候,PPT 已经增加到了将近 80 页。</p>
<p>到十月份,准备第四届 ESCC 的时候,这份 PPT 已经冗长到我自己也不再愿意用了。于是干脆重新写了一份《{{more}} Kibana 4》,也算是呼应了去年第三届 ESCC 时我讲的《{{more}} Kibana》话题。而这时候,已经有听众朋友线下跟我说:『全场分享嘉宾里你的气度最像一个讲师了』。</p>
<h2 id="section-2">代码</h2>
<p>和干的活类似,今年写的代码也都在这个领域,主要来说,写了一个 Kibana 4.2 的 visualize 扩展,叫 sankey chart。也是我 ESCC 演讲的主要实例。随后上海的分享后,medcl 说到场的 Elastic.co 的布道师团队负责人<a href="https://github.com/ycombinator">@ycombinator</a>觉得这个扩展不错,回去催促 kibana 团队加快对这个 <a href="https://github.com/elastic/kibana/pull/4832">pull request</a> 的 review。评论中,也还有好几位同好表示 “huge potential”, “a big +1”, “very useful”。可以说,这是做开源最幸福的事情啦~</p>
<p>另一个比较大的,是给 Rsyslog 提交的代码。在微博我们大规模运用了 Rsyslog 作为日志中转乃至数据处理的任务。从 Rsyslog 源码和测试用例集中发现了一些文档中都还没提及的用法可能性,也顺带就测试出来一些 bug。为 Rsyslog 新增了的 action.copymsg 选项,扩展了 omkafka 模块的 maxoutputqsize 性能数据统计项,新增了 mmgrok 和 mmdblookup 模块。当然,在交流中也发现了 Rsyslog 作为批量处理的缺点:Rsyslog 的设计逻辑是把数据尽快发出去,只有在发不出去的情况下,才会积累出队列批量处理。这跟 Elasticsearch 的优化路线是背道而驰的。Rsyslog 作者在社区呼声中表示会抽空提供一个队列控制的办法,不过预计短期内他是没空的……</p>
<p>另一方面自然还是继续关注 Perl。Perl6 终于在前天发布了!!上个月曾经尝试过用 Perl6 实现一个 Logstash,发现要实现到 Logstash 1.3 的语法支持度,基本上百行代码就够了。Perl6 的 Grammar 设计真的超方便。唯一的问题就是:性能性能性能!不知道明年这时候,Perl6 的性能会提升多少……</p>
<h2 id="section-3">生活</h2>
<p>六月借着去上海演讲的机会,去杭州休息了几天;十一月则趁离职休假,去西安休息了几天。相比来说,杭州是舒适的,可以安安静静的在西湖边上走走停停,一天就美好的过去;西安是厚重的,计划中一天看两个博物馆,压根就逛不完。老婆大人最后用一句话解决了心中的纠结和矛盾:『以后有小孩了肯定要带来看兵马俑的,还怕没机会再来西安么?』</p>
<p>然而一想:其实大多数曾经到过的城市,不会再有机会看一眼了。真的好伤怀……以 IT 宅男码农的身份,感觉只能期盼全国各地都赶紧出一些牛逼的互联网公司,然后才有机会了啊……创业者们,加油~</p>
<p>再看看北京现在这个雾霾天,真是更加想念那些美好的地方啊。</p>
<h2 id="section-4">发展</h2>
<p>临近年底,选择了离职。微博移动端运维是个具有很强战斗力的团队。几乎每一两个人就要,也做到了支撑起一个方向上的所有任务。我相信这是一个可以作为国内 SRE 建设典范的团队。但是作为已经在日志处理上耗费了将近两年时间的个人,思考再三决定试试看把这种深度的经验做个变现。这个决定还得感谢之前人人网的前同事和前领导们,虽然你们引诱我跳槽的目的失败了,但是你们说的道理我接受了。尽管最后我做的决定刚好相反,不是找个中小型公司转型带团队,而是彻底地扎进日志的无底洞……</p>
<p>明年或许不会像今年这样出没在各种大会小会上,但是对各种运维技术领域的知识的学习,不会也不能中断。虽然现在 devopsweekly 里十有五六都是 docker, docker, docker……但未来谁知道呢,~</p>
<p>最后,在朋友的邀请下,准备明年开始尝试一下做点小规模的线下面授培训的活动。话题自然是 ELK Stack 相关。以 ELK 的发展,或许明年就会有不少急缺高级 ELK 经验的岗位呢,到时候欢迎找我要人,哈哈~~</p>
<p>预祝 2016 年年终个人总结时,我会一如既往的对自己满意,对未来充满信心。</p>
Rsyslog 的 mmnormalize 模块用法
2015-11-25T00:00:00+08:00
rsyslog
http://chenlinux.com/2015/11/25/rsyslog-mmnormalize
<p>mmnormalize 是 Rsyslog 内置的一种数据解析的方案,甚至有自己的官网:<a href="http://www.liblognorm.com">http://www.liblognorm.com</a>可以阅读相关用法细节。它既不像 Rsyslog 的 rainerscript 那样采用 ERE 类型的简单正则,也不像 Logstash的 Grok 那样采用 PCRE 类型的复杂正则(一度通过添加 regex parser 引入过 PCRE,后来又删了),而是自己设计了一套方式,其最核心的匹配语法就是 <code class="highlighter-rouge">%char-to:</code> 这种“向后匹配直到*为止”。下面是一段解析 nginx 访问日志的 mmnormalize 配置,相信大家第一眼看上去都会晕:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>rule=:"%client_ip:char-to:"%" %tcp_peer_ip:ipv4% - [%req_time:char-to:]%] "%verb:word% %url:word% %protocol:char-to:"%" %status:interpret:int:number% %latency:interpret:float:word% %bytes_sent:interpret:int:number% "%referrer:char-to:"%" "%user_agent:char-to:"%" %upstream_addrs:tokenized:, :tokenized: \x3a :regex:[^ ,]+% %upstream_response_times:tokenized:, :tokenized: \x3a :interpret:float:regex:[^ ,]+% %pipe:word% \t %host:word% cache_%cache:word%
</code></pre>
</div>
<p>不过上个月 liblognorm 做了一次重大版本更新,新的 v2 语法,添加了一个 <strong>user-defined types</strong> 的设计,这就有点类似 Grok 的预定义正则的意思啦。</p>
<p>所以,本文来详细说说,从 v1.1.0 开始,新增的一些 liblognorm 的 type 给我们处理 Rsyslog 数据带来的便利。</p>
<p><em>normalize 的匹配规则叫做 rulebase,所以可以看到有些 rsyslog 介绍中,mmnormallize 配置文件的后缀名是 <code class="highlighter-rouge">*.rb</code>,可不要以为是用 Ruby 解析啊。</em></p>
<div class="highlighter-rouge"><pre class="highlight"><code>version=2
type=@TIMESTAMP:%date:date-iso% %time:time-24hr%Z
type=@TIMESTAMP:%datetime:date-rfc5424%
type=@TIME:%resptime:float%
type=@TIME:-
rule=:%timestamp:@TIMESTAMP% %clientip:ipv4% %resptime:@TIME% %urlpath:string% %reqbody:json% %referer:quoted-string%
</code></pre>
</div>
<p>这行 rule 在 v2 中还可以写的更美观一些:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>rule=:%[ {"type":"@TIMESTAMP", "name":"timestamp"},
{"type":"literal", "text:" "},
{"type":"ipv4", "name":"clientip"},
{"type":"literal", "text:" "},
{"type":"@TIME", "name":"resptime"},
{"type":"literal", "text:" "},
{"type":"string", "name":"urlpath"},
{"type":"literal", "text:" "},
{"type":"quoted-string", "name":"referer"},
{"type":"literal", "text:" "},
{"type":"json", "name":"reqbody"}
]%
</code></pre>
</div>
<p>Rsyslog 的 mmjsonparse 模块只能解析 CEE 格式,如果 msg 本身是纯 JSON 的,反而不能解析,这时候就可以用上 mmnormalize 的 json parser 了。</p>
<p>和 json 一样也是 v1.1 以后才加入的,还有 <code class="highlighter-rouge">char-sep</code> 和 <code class="highlighter-rouge">rest</code>,<code class="highlighter-rouge">char-sep</code> 和 <code class="highlighter-rouge">char-to</code> 的区别是前者是0到多个,后者是1到多个;<code class="highlighter-rouge">rest</code> 则用来收集当前位置到本行结尾的全部数据。</p>
<p>也就是说:<code class="highlighter-rouge">%capturename:char-sep:\x20</code> 等于 <code class="highlighter-rouge">%capturename:char-sep: %</code> 等于 <code class="highlighter-rouge">%{"type":"char-sep","name":"capturename","extradata":" "}%</code> 等于 <code class="highlighter-rouge">%capturename:char-sep{"extradata":" "}</code>。相当于 Grok 里的 <code class="highlighter-rouge">[^ ]*?</code>。其他类似。</p>
<p>如果觉得上面那种预定义太麻烦,毕竟响应时间无非就是数值或者横杆而已,那么这行还可以这么写:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="nt">"type"</span><span class="p">:</span><span class="s2">"alternative"</span><span class="p">,</span><span class="w">
</span><span class="nt">"parser"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="nt">"name"</span><span class="p">:</span><span class="s2">"resptime"</span><span class="p">,</span><span class="w"> </span><span class="nt">"type"</span><span class="p">:</span><span class="s2">"float"</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="nt">"type"</span><span class="p">:</span><span class="s2">"literal"</span><span class="p">,</span><span class="w"> </span><span class="nt">"text"</span><span class="p">:</span><span class="s2">"-"</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
</div>
<p>还有类似 logstash-filter-kv 插件的功能,比如把</p>
<div class="highlighter-rouge"><pre class="highlight"><code>a:2,b:4, c:6, d:8
</code></pre>
</div>
<p>这段数据做切割处理的配置:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>%{"name":"obj", "type":"repeat",
"parser":[
{"type":"string", "name":"key"},
{"type":"literal", "text":":"},
{"type":"number", "name":"val"}
],
"while": {
"type":"alternative", "parser": [
{"type":"literal", "text":", "},
{"type":"literal", "text":","}
]
}
}%
</code></pre>
</div>
<p>会解析得到下面这样的 JSON 结果:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nt">"obj"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nt">"val"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2"</span><span class="p">,</span><span class="w"> </span><span class="nt">"key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"a"</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nt">"val"</span><span class="p">:</span><span class="w"> </span><span class="s2">"4"</span><span class="p">,</span><span class="w"> </span><span class="nt">"key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"b"</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nt">"val"</span><span class="p">:</span><span class="w"> </span><span class="s2">"6"</span><span class="p">,</span><span class="w"> </span><span class="nt">"key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"c"</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nt">"val"</span><span class="p">:</span><span class="w"> </span><span class="s2">"8"</span><span class="p">,</span><span class="w"> </span><span class="nt">"key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"d"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
</div>
<p>看起来似乎不是很 kv 的样子,不过对于写入 Elasticsearch 来说,却刚刚好符合 nested object 的设计!</p>
<p>不过目前,还有些匹配模式在 v2 中不支持的,还得继续使用 v1 模式:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>rule=:%filesize:interpert:float:number%
</code></pre>
</div>
<p>有时候,明明你这个数据中是整形,但是因为 ES 的 mapping 问题或者其他原因,需要强制转换成浮点型。Rsyslog 本身的 rainerscript 只提供了 <code class="highlighter-rouge">cnum()</code> 函数,没有 <code class="highlighter-rouge">cfloat()</code>,那么我们只能在 mmnormalize 里做 interpert 转换了。而这个操作目前在 v2 版本中还不支持。</p>
<p>目前 liblognorm 所支持的所有匹配格式说明,见<a href="https://github.com/rsyslog/liblognorm/blob/master/doc/configuration.rst">https://github.com/rsyslog/liblognorm/blob/master/doc/configuration.rst</a></p>
SIREn 插件试用
2015-10-29T00:00:00+08:00
elasticsearch
http://chenlinux.com/2015/10/29/siren
<p>SIREn 是一个基于 Lucene 做的,专门针对 nested object 数据做优化的方案。其官网地址:<a href="http://siren.solutions">http://siren.solutions</a>。SIREn 自己并不提供完整的软件,而是以 Solr 或者 Elasticsearch 插件的形式存在。在 SIREn 官网首页写着,自己是 trush schemaless,high performance nested query。而我之前已经写博客说过,Elasticsearch 的 schemaless 是有限制的,同一个 index 下,field 的 mapping 是必须唯一一致的。否则,或者写入失败,或者搜索异常。</p>
<p>那么我们来试一下这个 SIREn 看看。首先是下载运行:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># wget http://siren.solutions/download/siren-elasticsearch-1.4-bin.zip</span>
<span class="c"># unzip siren-elasticsearch-1.4-bin.zip</span>
<span class="c"># cd siren-elasticsearch-1.4-bin</span>
<span class="c"># ./example/bin/elasticsearch</span>
</code></pre>
</div>
<p>然后我们尝试写入几条 mapping 有冲突的数据:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># curl -XDELETE "http://localhost:9200/napr"</span>
<span class="c"># curl -XPOST "http://localhost:9200/napr"</span>
<span class="c"># curl -XPUT "http://localhost:9200/napr/chargepoint/_mapping" -d '</span>
<span class="o">{</span>
<span class="s2">"chargepoint"</span> : <span class="o">{</span>
<span class="s2">"properties"</span> : <span class="o">{</span>
<span class="s2">"_siren_source"</span> : <span class="o">{</span>
<span class="s2">"analyzer"</span> : <span class="s2">"concise"</span>,
<span class="s2">"postings_format"</span> : <span class="s2">"Siren10AFor"</span>,
<span class="s2">"store"</span> : <span class="s2">"no"</span>,
<span class="s2">"type"</span> : <span class="s2">"string"</span>
<span class="o">}</span>
<span class="o">}</span>,
<span class="s2">"_siren"</span> : <span class="o">{}</span>
<span class="o">}</span>
<span class="o">}</span><span class="s1">'
# curl -XPUT "http://localhost:9200/napr/chargepoint/1" -d '</span>
<span class="o">{</span>
<span class="s2">"ChargeDeviceName"</span>: <span class="s2">"1c Design Limited, Glasgow (1)"</span>,
<span class="s2">"Accessible24Hours"</span>: <span class="nb">false</span>
<span class="o">}</span><span class="s1">'
# curl -XPUT "http://localhost:9200/napr/chargepoint/2" -d '</span>
<span class="o">{</span>
<span class="s2">"ChargeDeviceName"</span>: <span class="s2">"2c Design Limited, Glasgow (2)"</span>,
<span class="s2">"Accessible24Hours"</span>: <span class="s2">"true"</span>
<span class="o">}</span><span class="s1">'
# curl -XPUT "http://localhost:9200/napr/chargepoint/3" -d '</span>
<span class="o">{</span>
<span class="s2">"ChargeDeviceName"</span>: <span class="s2">"3c Design Limited, Glasgow (3)"</span>,
<span class="s2">"Accessible24Hours"</span>: 123
<span class="o">}</span><span class="s1">'
# curl -XPUT "http://localhost:9200/nepr/chargepoint/4" -d '</span>
<span class="o">{</span>
<span class="s2">"ChargeDeviceName"</span>: <span class="s2">"4c Design Limited, Glasgow (4)"</span>,
<span class="s2">"Accessible24Hours"</span>: <span class="o">[</span>123, 234, 345, 456]
<span class="o">}</span><span class="s1">'
</span></code></pre>
</div>
<p>ok,三条数据都写入成功了。</p>
<p>然后我们用原始的 Elasticsearch 语法尝试去获取『大于100』的数据:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># curl -XPOST "http://localhost:9200/nepr/_search?q=Accessible24Hours:>100"</span>
<span class="o">{</span><span class="s2">"took"</span>:16,<span class="s2">"timed_out"</span>:false,<span class="s2">"_shards"</span>:<span class="o">{</span><span class="s2">"total"</span>:5,<span class="s2">"successful"</span>:5,<span class="s2">"failed"</span>:0<span class="o">}</span>,<span class="s2">"hits"</span>:<span class="o">{</span><span class="s2">"total"</span>:0,<span class="s2">"max_score"</span>:null,<span class="s2">"hits"</span>:[]<span class="o">}}</span>
</code></pre>
</div>
<p>可以看到,搜索结果是空。</p>
<p>而用 SIREn 的树状结构语法获取:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># curl -XPOST "http://localhost:9200/nepr/_search" -d '</span>
<span class="o">{</span>
<span class="s2">"query"</span>: <span class="o">{</span>
<span class="s2">"tree"</span> : <span class="o">{</span>
<span class="s2">"node"</span> : <span class="o">{</span>
<span class="s2">"attribute"</span> : <span class="s2">"Accessible24Hours"</span>,
<span class="s2">"query"</span> : <span class="s2">"xsd:long([100 TO *])"</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span><span class="s1">'
{"took":29,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":2,"max_score":4.0,"hits":[{"_index":"nepr","_type":"chargepoint","_id":"4","_score":4.0,"_source":
{
"ChargeDeviceName": "4c Design Limited, Glasgow (4)",
"Accessible24Hours": [123, 234, 345, 456]
}},{"_index":"nepr","_type":"chargepoint","_id":"3","_score":1.0,"_source":
{
"ChargeDeviceName": "3c Design Limited, Glasgow (3)",
"Accessible24Hours": 123
}}]}}%
</span></code></pre>
</div>
<p>yes,我们拿到了这条数据!</p>
<p>更复杂一点,我们再来:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># curl -XPOST "http://localhost:9200/nepr/_search" -d '</span>
<span class="o">{</span>
<span class="s2">"query"</span>: <span class="o">{</span>
<span class="s2">"tree"</span> : <span class="o">{</span>
<span class="s2">"node"</span> : <span class="o">{</span>
<span class="s2">"attribute"</span> : <span class="s2">"Accessible24Hours"</span>,
<span class="s2">"range"</span> : <span class="o">[</span>2,3],
<span class="s2">"query"</span> : <span class="s2">"xsd:long([10 TO *])"</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>,
<span class="s2">"aggs"</span>: <span class="o">{</span>
<span class="s2">"1"</span>: <span class="o">{</span>
<span class="s2">"terms"</span>: <span class="o">{</span>
<span class="s2">"field"</span>: <span class="s2">"ChargeDeviceName"</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span><span class="s1">'
</span></code></pre>
</div>
<p>这里添加了一个 <code class="highlighter-rouge">range</code> 选项,SIREn 对所有的数组默认就做 nested 处理了,所有是有序的。这个选项的意思就是,只对数组中第 2 到 3 位节点的数据做搜索请求。这下,搜索结果变成了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="nt">"took"</span><span class="p">:</span><span class="mi">9</span><span class="p">,</span><span class="nt">"timed_out"</span><span class="p">:</span><span class="kc">false</span><span class="p">,</span><span class="nt">"_shards"</span><span class="p">:{</span><span class="nt">"total"</span><span class="p">:</span><span class="mi">5</span><span class="p">,</span><span class="nt">"successful"</span><span class="p">:</span><span class="mi">5</span><span class="p">,</span><span class="nt">"failed"</span><span class="p">:</span><span class="mi">0</span><span class="p">},</span><span class="nt">"hits"</span><span class="p">:{</span><span class="nt">"total"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"max_score"</span><span class="p">:</span><span class="mf">2.0</span><span class="p">,</span><span class="nt">"hits"</span><span class="p">:[{</span><span class="nt">"_index"</span><span class="p">:</span><span class="s2">"nepr"</span><span class="p">,</span><span class="nt">"_type"</span><span class="p">:</span><span class="s2">"chargepoint"</span><span class="p">,</span><span class="nt">"_id"</span><span class="p">:</span><span class="s2">"4"</span><span class="p">,</span><span class="nt">"_score"</span><span class="p">:</span><span class="mf">2.0</span><span class="p">,</span><span class="nt">"_source"</span><span class="p">:</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nt">"ChargeDeviceName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"4c Design Limited, Glasgow (4)"</span><span class="p">,</span><span class="w">
</span><span class="nt">"Accessible24Hours"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="mi">123</span><span class="p">,</span><span class="mi">234</span><span class="p">,</span><span class="mi">345</span><span class="p">,</span><span class="mi">456</span><span class="p">]</span><span class="w">
</span><span class="p">}}]},</span><span class="nt">"aggregations"</span><span class="p">:{</span><span class="nt">"1"</span><span class="p">:{</span><span class="nt">"buckets"</span><span class="p">:[{</span><span class="nt">"key"</span><span class="p">:</span><span class="s2">"4"</span><span class="p">,</span><span class="nt">"doc_count"</span><span class="p">:</span><span class="mi">1</span><span class="p">},{</span><span class="nt">"key"</span><span class="p">:</span><span class="s2">"4c"</span><span class="p">,</span><span class="nt">"doc_count"</span><span class="p">:</span><span class="mi">1</span><span class="p">},{</span><span class="nt">"key"</span><span class="p">:</span><span class="s2">"design"</span><span class="p">,</span><span class="nt">"doc_count"</span><span class="p">:</span><span class="mi">1</span><span class="p">},{</span><span class="nt">"key"</span><span class="p">:</span><span class="s2">"glasgow"</span><span class="p">,</span><span class="nt">"doc_count"</span><span class="p">:</span><span class="mi">1</span><span class="p">},{</span><span class="nt">"key"</span><span class="p">:</span><span class="s2">"limited"</span><span class="p">,</span><span class="nt">"doc_count"</span><span class="p">:</span><span class="mi">1</span><span class="p">}]}}}</span><span class="err">%</span><span class="w">
</span></code></pre>
</div>
<p>可以看到,因为 <code class="highlighter-rouge">_id</code> 为 3 的文档里 Accessible24Hours 字段只有一个值,所以无法匹配上从第二个值开始的多个值的对比,也就没被过滤出来了。</p>
<hr />
<p>不过 SIREn 目前比较尴尬的是,他只基于 ES 做了 query 部分,aggregation 部分还是老样子,必须类型一致才行,这也导致 SIREn 示例文件数据里把一些冲突日志去掉了的原因。</p>
<p>如果使用的是 Solr,SIREn 插件的做法是只定义两个 field,一个是 UUID,一个是 JSON。然后 siren 处理的所有数据存在这个 JSON 字段里(类似 ES 插件里的那个 <code class="highlighter-rouge">_siren_source</code> 字段)。这也就能达到全部 JSON schemaless。此外,SIREn 的 Solr 插件还实现了 nested facet 支持,也可以尝试。</p>
<p>总之,SIREn 扩展采用树形方式自行处理一个在 ES、Solr 看来多出来的字段,而并不影响原有字段的处理流程。所以,这对 ES 有几个影响:</p>
<ul>
<li>其他字段还是会判断数据类型并生成 mapping,所以写入依然会有问题。</li>
<li>aggregation 还是走 ES 的实现,导致根据 number 过滤出来的文档,在 aggregation 时却会按照 boolean(即 mapping 中的记录)检测,aggregation 请求直接报错不计算。</li>
<li>重复一遍树状索引数据,导致膨胀率翻倍增高。实测,一段大小约为 30MB 的数据,在 ES 默认环境中会膨胀到 50MB,而在开启 SIREn 插件的环境下则膨胀到了 120MB!</li>
</ul>
ESCC 参会笔记
2015-10-25T00:00:00+08:00
elasticsearch
kibana
http://chenlinux.com/2015/10/25/escc
<p>2015 年 10 月 25 日,ESCC 2015 上海站召开,感谢携程的大力支持,让我得以参与,参会笔记如下:</p>
<p>上海站的分享,和北京站集中在 ELK 经验分享不太一样,各个方面、层面都有涉及。</p>
<p>上午,分别是 ES 2.0 介绍和 Logstash 2.0 介绍。都是 ES 原厂工程师的英文演讲,以个人的感觉,口音听起来还是蛮舒服能听懂的。</p>
<p>ES 2.0 的主要特性和更新其实在官方博客上陆陆续续大多是提过了的。不过在 mapping 冲突的示例上,我觉得这次演讲选择的更好:举例的是相同 type,不同 analyzer 的冲突,比官博上不同 type 的来说,更明确,不会让人误以为只要 type 一致就算 mapping 一致了。</p>
<p>演讲后尝试问了两个问题,一个是想到曾经看到一个<a href="https://github.com/elastic/elasticsearch/issues/10032">issue</a>,里面提到可以对不再写入数据的索引关掉 IndexWriter 节省资源,所以询问这个事情有没有进展,imotov 回答说他不记得具体有这个 issue,但是在较高的 Lucene 版本中这个 IndexWriter 占用的资源是跟实际有没有写入操作相关的,所以从 ES 1.5 版以后,应该这个 IndexWriter 开着不会是什么问题。我回来翻了一下 issue,原来这个已经 close 了,官方是有选择的解决这个问题,对应的是 <a href="https://github.com/elastic/elasticsearch/pull/11336">synced flush</a>,在这里解释了不搞一个 read only mode 的理由:更灵活并且方便轻量级的自动化操作。synced flush 大家都知道了,在 ES 1.6 的时候已经有了。</p>
<p>另一个问题是 ES 的 dynamic script 支持。我们知道 ES 从 1.4.3 开始关掉了 groovy sandbox 的 dynamic script 支持,改用了只支持数值操作的 lucene expression。当时的公告中,说的是 ES 开发组会跟 Lucene 开发组一起努力加强 expression 的功能。对这个问题,imotov 的回答是:很遗憾,目前没有,因为这不是单方面能决定的。然后在下午我看 github 的时候,发现 elastic 组织下最近这周刚多了一个项目,叫:<a href="https://github.com/elastic/Painless">plan A</a>。它的描述是:”New Scripting Language for ElasticSearch”!不过目前没啥内容,感觉可以期待~</p>
<p>Logstash 2.0 的演讲,或许因为早先设想的高级特性太多太好,然而基本都延期了,2.0 里不会有,所以没太多可期待的……只能说继续等待 2.1 或者 2.2 。</p>
<p>接着是小排虞冰的分享。用 ES 做核心业务的数据支持,真的是一直比较少的外部经验分享。虽然和 ELK 的日志场景不太一致,导致优化手段和方向甚至就几乎相反,但是作为一个比较通用的后端服务架构设计,依然是一个有价值的分享。演讲也对 ES 的维护和推广提了一点看法——比如:用 QueryBuilder 给 ES 实现 ORM;以及需要至少一个以上的 ES 熟练工才能上业务线。</p>
<p>下午是自己的第一个分享,稍微有点小尴尬的是 ppt 不太新,好在显示屏给力,还是正常完成的。收到三个问题,前两个其实类似(单 panel 时间段,同比环比 panel),都是在 Vis 里不能单独地固定 time filter,从目前 Kibana4 的设计逻辑上,确实是没办法了。最后一个是 index pattern 的通配符问题,后来经过查阅 kibana4 代码实现,这块应该是支持的。</p>
<p>然后是 wood 大叔的演讲,虽然大多数优化手段之前在 QQ 群里都说过了,但是 wood 把一些原理性的东西说的很清楚,整个演讲听下来还是对一些细节有了更新的认识。</p>
<p>茶歇中,做了一下抽奖活动,把出版社给我准备的十本书都发出去了,这次采用了一个比较新奇的方法:给主持人手机打电话,谁能抢先拨进去,谁就算中奖!事后收到有微信说:好可惜没抽上的,已经下单购买了。哈哈~</p>
<p>最后 medcl 演讲,里面提到的有一点是第一手新闻:packetbeat 不单单可以作为像 tcpdump 这样的方式运行,还可以以 app 的方式运行。在列举 beats 家族成员的时候,更是列出了一个目前在 github 的 elastic 用户下面还没立项的秘密项目。可谓是个惊喜。</p>
<p>本来在下午休息的时候,我还临时写了 11 页的小 ppt,打算闪电演讲的时候起个头。不过看大家都没有参与的意思,也就没说出来。或许上海的 conference 还是不太多,大家显得都不是那么活跃。这次聚会最终到场 100 人左右,毕竟是第一次在上海举办,可能也相互比较默认和拘谨吧。QQ 群聊天跟实际还是有差距的,相信明年,肯定会更好!</p>
ESCC 参会笔记
2015-10-20T00:00:00+08:00
elasticsearch
kibana
http://chenlinux.com/2015/10/20/escc
<p>10 月 17 号举办了第四届 ESCC(ElasticSearch China Conference) 北京站。作为个人习惯,稍作记录。</p>
<p>今年场地换到了中科院软件所。之前曾经参加过一次 openstack 的活动,也是在这个场地,不过这次居然还提供了午餐,软件所对开源社区的支持真的是蛮积极的。会前在 QQ 群里就看到,甚至有从成都上海赶过来的同好(他们开始不知道自己城市稍后也会有)。</p>
<p>清早起来,和去年 ESCC 一样又是一个雾霾天……和去年相比,今年的签到处显得正式多了:有签名墙,有易拉宝和传单,还有各色小贴纸派送~尤其是 Found.io 的,原先只见过小狗造型,这次见到三四种,漂亮死了~按 medcl 所说,这批小礼品是 elastic.co 公司花了上千元快递费从国外特意送过来的!</p>
<p>同样特意送来的还有 ES 作者 Shay 录制的对 ESCC 的祝福视频。Shay 特意讲到了他在发布 ES 时收到的第一个评论,对方用 ES 来支持对抗流行疾病的斗争,这是一种极大的激励。我想:如果上帝为了遏制人类而发明这么多语言,那么开源运动就是新时代人类的通天塔吧!</p>
<p>会议另一个巧合是,两位讲师,凌霄和刘波涛,穿了同款 T shirt,都是一只麋鹿头像的文化衫。虽然麋鹿图标在 ELK 正式产品上没出现多久,但是 ELK 麋鹿形象看来还是深入人心了~</p>
<p>和去年 ESCC 相比,今年的分享主题有很大不同。去年除了我讲 Kibana,其他讲师都是在分享 search 和 score 相关的话题。今年除了 medcl,都在分享 ELK。所以会后有参会人员评论说:去年想听优化听不到,今年全在讲优化然而用不着了…… anyway,从各司经验来看,采用 Kafka 作为 broker 角色,采用 Java 自开发程序作为 Kafka 和 Elasticsearch 之间的 indexer 角色,几乎成为通用的海量场景优化方式。可惜的时候,没有人具体给解释为什么我们要放弃 Logstash 转而自研?为什么用的是 Java?其实这是一个蛮有趣的话题,在我演讲结束的提问环节,有朋友提问在 flume/fluentd/logstash/rsyslog 之间怎么做最优选择,可惜时间不够,我除了表明我使用 rsyslog 也有条件所限的原因外,没能铺开来讲。在博客上可以多说几句了:</p>
<ul>
<li>Logstash 采用 JRuby 语言,其中处理时间,连接 kafka 和 Elasticsearch 的三个环节,都是通过 <code class="highlighter-rouge">require "java"; java_import ***</code> 的方式加载的。这个过程中,Ruby 到 Java 的类型转换等等,都是会影响到性能的。去年我曾经测试过用 JRuby 来加载 netty 库实现一个极简的 TCPServer,每秒只能到 5w 的处理能力,这基本上跟直接用 Ruby 默认 socket 库的效果是一致的。</li>
<li>logstash-input-kafka 插件默认采用 json codec,会在<strong>单线程</strong>中调用 jrjackson 库(注意这里同样有 JRuby 的损耗)做序列化。在 slideshare 上,linkedin 的分享说明他们宁愿采用 logstash-input-pipe 调用 kafka-consumer-console.sh 脚本;在 discuss 中,也有人推荐修改 codec 为 line,然后多线程运行 logstash-filter-json 来解析数据。根据 QQ 群里金桥童鞋的测试,能提高一倍的性能。</li>
</ul>
<p>所以说,Logstash 选用 JRuby,是为了在不失灵活性的前提下,尽可能方便的接入各种现存系统。而不是纯粹为了提高性能(那会儿 jordansissel 还没见过 kafka 呢,他只考虑过 joda 库比 Ruby 的 time 库快的问题)。</p>
<p>回到演讲。百度高攀的分享中,给我们展示了一个很有趣的做法:当你机器内存又大,磁盘又多,还有 SSD 空闲的时候,怎么有效利用起来?这个话题,在前两个月,携程的朋友也有类似的分享,我有邀稿分享在我的书中,区别在于:携程的 SSD 节点和 SAS 节点是不同机器,索引由热变凉时,需要走网络迁移;百度的 SSD 节点和 SAS 节点是在同一台机器上。分层存储,其实是前几年很热的一个话题,flashcache 一度是很多数据库优化的必备步骤。有环境条件的童鞋其实也可以一试。</p>
<p>高攀提到另一个改进,有关 recovery 期间增量 translog 的处理,没太听清他的改进方案。以我目前的理解力,感觉官方方案应该不会有特明显的卡顿影响,或许是规模还没到的原因。</p>
<p>Admaster 宋兵强的分享,是我个人感觉这次 ESCC 最有意思的一段,可以看出他是长期做大数据处理的,从他的历史经验,推导出来一些他对 ES 未来发展面临的问题和方向的猜测,非常有意思!而在提问环节,一位百度的工程师站起来第一句话是:宋师兄好,我看过你在百度时的代码。非常触动我,互联网是个跳槽非常频繁的行业,铁打的营盘流水的兵,能以这种代码识人,真是幸甚至哉!</p>
<p>这个环节中,最让大家 happy 的是 medcl 补充了一个消息:marvel 在不远的未来就会完全免费使用啦!</p>
<p>中午,吴怡编辑驱车数十公里赶来给我把几本样书送到了。为了赶在聚会之前上架,我催她好些回,再次感谢她的理解和支持!</p>
<p>下午芒果TV 刘波涛的分享,比较好玩的地方在于他们对多机房之间数据传输的考量。不确定是不是受我影响,也采用了较多的 rsyslog。加上近来在 rsyslog 社区里,绝大多数提问也都跟 Elasticsearch 相关,我都考虑是不是接下来再写一本《Rsyslog 指南》的 gitbook 了。</p>
<p>我自己演讲完后,实在坚持不住,趴着小睡了会儿,medcl 对不住,不是你讲的不好,是我自己有午睡习惯……</p>
<p>biglog 张磊的分享带上了 demo 录像展示,很棒!对于很多只听说过 ELK 不知道有啥用的人来说,一图胜千言,一视频胜千图!然后其中对 suricata、ntop 的介绍也非常不错。分享中另一个重点在跨数据中心的集群方式。我开始猜测他指的会是采用 tribe 节点来串联不同集群给 Kibana 做查询。结果不是!而是利用 allocate routing 的 zone 设置,把对应机房的索引分片分配在自己机房的节点上,以保证不跨网传输。同时又采用 elasticsearch-zookeeper 来避免 elasticsearch 自身的跨机房 discovery 流量,维护集群稳定。确实是一个我完全没想到的玩法~~</p>
<p>附注:ntop-ng 有一个自己的 Kibana3 fork,叫 Qbana,我在书中有提过。ntop 的 PF_RING 抓包方式,在 elastic 的 packetbeat 中也有采用。未来 ELK 肯定会有对这个方向更好的支持。</p>
<p>最后闪电演讲。我从 2011 年第一次在 Perlconf 接触这个形式后就喜欢上了,所以提前跟团队的伙伴提及,可以选个有趣的小话题聊一下。效果来看,感觉炳哲做的不错,赞!</p>
<p>最后大轴是一位老先生,之前在张磊的分享中,他就对 PF_RING 方式提出来质疑,然后闪电演讲的时候,亲自上场,介绍了他自己基于 DPDK 的实现方式。这种友好的气氛,真是太让人喜欢了~哈哈</p>
<hr />
<p>也说一些我个人觉得还不是特到位的地方:</p>
<p>两次抽奖送书环节,都是先宣布休息才抽,场面上已经乱了。其实拍个中奖者合影啊什么的,再宣布休息,可能会更好一些。<br />
闪电演讲环节,静悄悄的在会议安排中添加,静悄悄的在最后就开始了,没有燃烧全场的那种激情。或许应该在早上开始大会之前,统一给大家介绍一下这个环节,然后留一整天时间做现场报名,可能会有更好的效果(yes,我说的就是上面那种友好的质疑)。</p>
<p>不论如何,这是一次成功的,近乎完美的大会。感谢 elastic.co,感谢 medcl,感谢讲师们,感谢志愿者们,感谢全体参会同仁们!</p>
rsyslog 中 if 条件判断的限制
2015-09-24T00:00:00+08:00
logstash
linux
rsyslog
http://chenlinux.com/2015/09/24/condition-in-rsyslog
<p>Rsyslog 从 v6 以后,实现了全新的 rainerscript 语法,数据处理灵活度大大提高。我最近一直在把 logstash 的解析配置迁移到 rsyslog 中完成。结果今天碰到一个非常好玩的地方。由此也说明了,一切 DSL,都不要想当然的觉得它会有跟编程语言完全一样的行为。</p>
<p>事情是这样的:一段 JSON 日志,在 rsyslog 中经过下面一段逻辑:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">set</span> <span class="nv">$</span><span class="err">!</span><span class="nv">datetime</span> <span class="o">=</span> <span class="nv">exec_template</span><span class="p">(</span><span class="s">"get_now_time"</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$</span><span class="err">!</span><span class="nv">msg</span><span class="o">!</span><span class="nv">date</span> <span class="p">)</span> <span class="k">then</span> <span class="p">{</span>
<span class="k">reset</span> <span class="nv">$</span><span class="err">!</span><span class="nv">datetime</span> <span class="o">=</span> <span class="nv">replace</span><span class="p">(</span><span class="nv">$</span><span class="err">!</span><span class="nv">msg</span><span class="o">!</span><span class="nv">date</span><span class="p">,</span> <span class="s">" "</span><span class="p">,</span> <span class="s">"T"</span><span class="p">)</span> <span class="o">&</span> <span class="s">"+0800"</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$</span><span class="err">!</span><span class="nv">msg</span><span class="o">!</span><span class="nv">video_time_duration</span> <span class="p">)</span> <span class="k">then</span> <span class="p">{</span>
<span class="nv">set</span> <span class="nv">$</span><span class="err">!</span><span class="nv">msg</span><span class="o">!</span><span class="nv">video_duration_num</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="nv">set</span> <span class="nv">$</span><span class="err">!</span><span class="nv">msg</span><span class="o">!</span><span class="nv">video_duration_timesum</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="nv">set</span> <span class="nv">$</span><span class="err">!</span><span class="nv">msg</span><span class="o">!</span><span class="nv">video_first_duration</span> <span class="o">=</span> <span class="nv">cnum</span><span class="p">(</span><span class="nv">$</span><span class="err">!</span><span class="nv">msg</span><span class="o">!</span><span class="nv">video_time_duration</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">!</span><span class="nv">duration</span><span class="p">);</span>
<span class="k">foreach</span> <span class="p">(</span> <span class="nv">$</span><span class="err">.</span><span class="nv">item</span> <span class="nv">in</span> <span class="nv">$</span><span class="err">!</span><span class="nv">msg</span><span class="o">!</span><span class="nv">video_time_duration</span> <span class="p">)</span> <span class="k">do</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$</span><span class="err">.</span><span class="nv">item</span><span class="o">!</span><span class="nv">type</span> <span class="o">==</span> <span class="s">"1"</span> <span class="p">)</span> <span class="k">then</span> <span class="p">{</span>
<span class="k">reset</span> <span class="nv">$</span><span class="err">!</span><span class="nv">msg</span><span class="o">!</span><span class="nv">video_duration_num</span> <span class="o">=</span> <span class="nv">$</span><span class="err">!</span><span class="nv">msg</span><span class="o">!</span><span class="nv">video_duration_num</span> <span class="o">+</span> <span class="mi">1</span><span class="p">;</span>
<span class="k">reset</span> <span class="nv">$</span><span class="err">!</span><span class="nv">msg</span><span class="o">!</span><span class="nv">video_duration_timesum</span> <span class="o">=</span> <span class="nv">$</span><span class="err">!</span><span class="nv">msg</span><span class="o">!</span><span class="nv">video_duration_timesum</span> <span class="o">+</span> <span class="nv">cnum</span><span class="p">(</span><span class="nv">$</span><span class="err">.</span><span class="nv">item</span><span class="o">!</span><span class="nv">duration</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$</span><span class="err">!</span><span class="nv">msg</span><span class="o">!</span><span class="nv">video_duration_num</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">)</span> <span class="k">then</span> <span class="p">{</span>
<span class="nv">unset</span> <span class="nv">$</span><span class="err">!</span><span class="nv">msg</span><span class="o">!</span><span class="nv">video_duration_num</span><span class="p">;</span>
<span class="nv">unset</span> <span class="nv">$</span><span class="err">!</span><span class="nv">msg</span><span class="o">!</span><span class="nv">video_duration_timesum</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>数据中,<code class="highlighter-rouge">date</code> 是一个 String ,而 <code class="highlighter-rouge">video_time_duration</code> 是一个 Array。但是实际运行起来,发现输出的数据里,根据 <code class="highlighter-rouge">date</code> 处理得到了 <code class="highlighter-rouge">datetime</code> 新字段,却完全没有 <code class="highlighter-rouge">video_first_duration</code>, <code class="highlighter-rouge">video_duration_num</code> 和 <code class="highlighter-rouge">video_duration_timesum</code> 等新字段的踪影。</p>
<p>看来 rsyslog 里的条件判断是不能针对 Array 做判断了,于是我又改成下面这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">if</span> <span class="p">(</span> <span class="nv">$</span><span class="err">!</span><span class="nv">msg</span><span class="o">!</span><span class="nv">video_time_duration</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">!</span><span class="nv">duration</span> <span class="p">)</span> <span class="k">then</span> <span class="p">{</span>
</code></pre>
</div>
<p>这样获取的就是一个实际的 String 内容了。但是实际运行起来,输出数据里,不但没有应该被处理出来的新字段,反而还多了一段:<code class="highlighter-rouge">, "video_time_duration[0]!duration" : { }, </code>!</p>
<p>这就有点像 Perl5 里的 exists 指令在判断多层哈希键的时候的行为了,不存在的键先自动创建出来……但是:rsyslog 现在在 if 条件判断里用数组下标获取数据的时候,居然把整段认为是一个 key 的内容,实在是无奈了……</p>
<p>最后,这里只能上最原始的办法了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">if</span> <span class="p">(</span> <span class="nv">$msg</span> <span class="nv">contains</span> <span class="s">"video_time_duration"</span> <span class="p">)</span> <span class="k">then</span> <span class="p">{</span>
</code></pre>
</div>
<p>以上。</p>
【翻译】Kibana 字段的自定义展示格式开发
2015-08-25T00:00:00+08:00
logstash
kibana
http://chenlinux.com/2015/08/25/kibana-custom-field-formatters
<p>原文地址:<a href="http://www.elasticsearch.org/blog/kibana-custom-field-formatters">http://www.elasticsearch.org/blog/kibana-custom-field-formatters</a></p>
<p>Kibana 4.1 引入了一个新特性叫字段展示格式(field formatters),让我们可以实时转换字段内容成更形象的样式。这个特性帮助我们不修改数据的存储方式,而用另一种方式显示它。有关 field formatters 的介绍,可以阅读之前一篇<a href="https://www.elastic.co/blog/kibana-4-1-field-formatters">博客</a>。</p>
<p>本文的目的,则是带大家过一遍 field formatters 的开发流程。从 field formatter 接口开始,自己实现一个基础的 formatter,可以字段给 error 单词加高亮效果,最后完成整个解决方案。</p>
<h2 id="section">起步</h2>
<p>Kibana 开发环境的搭建介绍可以在 <a href="https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#development-environment-setup">Kibana repository</a> 看到。</p>
<p>从 Kibana 根目录触发,field formatters 相关代码存在 <code class="highlighter-rouge">/src/ui/public/stringify</code> 目录下。目录结构如下所示:</p>
<p>/stringify<br />
|–type //包括各种 formatter<br />
|–icons<br />
|–editors //formatter 用来请求和显示附加信息的 HTML 页面<br />
|–<strong>tests</strong><br />
|–register.js //每个 formatter 都要在这里面注册</p>
<p>Kibana 4.1 里,formatters 位置则在 <code class="highlighter-rouge">/src/kibana/components/stringify</code>。如果你是看的 4.1 版,可能跟本文讲的路径稍有区别,请自动对应查找一下,本文以 git master 为准。</p>
<p>现在,让我们在 <code class="highlighter-rouge">type</code> 目录 下创建一个文件叫 <code class="highlighter-rouge">Highlight.js</code>,下面是初始代码:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nx">define</span><span class="p">(</span><span class="kd">function</span> <span class="p">(</span><span class="nx">require</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="kd">function</span> <span class="nx">HighlightFormatProvider</span><span class="p">(</span><span class="nx">Private</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">_</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'lodash'</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">FieldFormat</span> <span class="o">=</span> <span class="nx">Private</span><span class="p">(</span><span class="nx">require</span><span class="p">(</span><span class="s1">'ui/index_patterns/_field_format/FieldFormat'</span><span class="p">));</span>
<span class="nx">_</span><span class="p">.</span><span class="kr">class</span><span class="p">(</span><span class="nx">Highlight</span><span class="p">).</span><span class="nx">inherits</span><span class="p">(</span><span class="nx">FieldFormat</span><span class="p">);</span>
<span class="kd">function</span> <span class="nx">Highlight</span><span class="p">(</span><span class="nx">params</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">Highlight</span><span class="p">.</span><span class="nx">Super</span><span class="p">.</span><span class="nx">call</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="nx">params</span><span class="p">);</span>
<span class="p">}</span>
<span class="nx">Highlight</span><span class="p">.</span><span class="nx">id</span> <span class="o">=</span> <span class="s1">'highlight'</span><span class="p">;</span>
<span class="nx">Highlight</span><span class="p">.</span><span class="nx">title</span> <span class="o">=</span> <span class="s1">'Highlight'</span><span class="p">;</span>
<span class="nx">Highlight</span><span class="p">.</span><span class="nx">fieldType</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'string'</span><span class="p">];</span>
<span class="nx">Highlight</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">_convert</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">text</span><span class="p">:</span> <span class="nx">_</span><span class="p">.</span><span class="nx">escape</span><span class="p">,</span>
<span class="na">html</span><span class="p">:</span> <span class="nx">_</span><span class="p">.</span><span class="nx">escape</span>
<span class="p">};</span>
<span class="k">return</span> <span class="nx">Highlight</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">});</span>
</code></pre>
</div>
<p>每种字段格式,都实现为扩展 FieldFormat 的类。<code class="highlighter-rouge">Highlight.id</code> 用在 Kibana 内部跟踪 formatter,每个 formatter 必须采用不同的 id。<code class="highlighter-rouge">Highlight.title</code> 显示在 formatter 下拉选择框里,<code class="highlighter-rouge">Highlight.fieldType</code> 则描述自己适用于哪种类型的字段内容。</p>
<p><code class="highlighter-rouge">Highlight.prototype._convert</code> 是实际进行格式化的地方。包括有 text 和 html 两种方法。text 方法用于 tooltips, filters, legends, 和 axis markers。html 方法用于搜索表格内。两者都接收字段内容为输入,输出我们希望的展示内容。如果两个方法是一样的,可以直接赋值 <code class="highlighter-rouge">Highlight.prototype._convert</code> 为一个函数。给 error 单词加高亮的代码如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nx">Highlight</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">_highlight</span> <span class="o">=</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">val</span><span class="p">,</span> <span class="nx">replace</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">_</span><span class="p">.</span><span class="nx">escape</span><span class="p">(</span><span class="nx">val</span><span class="p">).</span><span class="nx">replace</span><span class="p">(</span><span class="sr">/</span><span class="se">(</span><span class="sr">error</span><span class="se">)</span><span class="sr">/g</span><span class="p">,</span> <span class="nx">replace</span><span class="p">);</span>
<span class="p">};</span>
<span class="nx">Highlight</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">_convert</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">text</span><span class="p">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">val</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">_highlight</span><span class="p">(</span><span class="nx">val</span><span class="p">,</span> <span class="kd">function</span> <span class="nx">convertToUpperCase</span><span class="p">(</span><span class="nx">match</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">match</span><span class="p">.</span><span class="nx">toUpperCase</span><span class="p">();</span>
<span class="p">});</span>
<span class="p">},</span>
<span class="na">html</span><span class="p">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">val</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">_highlight</span><span class="p">(</span><span class="nx">val</span><span class="p">,</span> <span class="s1">'<mark>$&</mark>'</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">};</span>
</code></pre>
</div>
<p>只要字段内容中有 error 文本字样,我们就会根据 HTML 或者 text 场景选择包含进 mark 元素或者是转换成大写形式。注意这里使用的 <code class="highlighter-rouge">_.escape(val)</code> 语句,这句可以用来放置 HTML 注入和跨站脚本攻击。</p>
<p>然后需要注册这个新的 field formatter。在 register.js 里添加:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nx">fieldFormats</span><span class="p">.</span><span class="nx">register</span><span class="p">(</span><span class="nx">require</span><span class="p">(</span><span class="s1">'ui/stringify/types/Highlight'</span><span class="p">));</span>
</code></pre>
</div>
<p>未来,我们(Kibana 开发组)可能会把这个功能以插件形式提供,届时注册方法会更加简单。</p>
<p>现在我们可以对 string 类型的字段选择 Highlight 作为 field formatter 了!</p>
<p><img src="https://www.elastic.co/assets/blt3b40cdcf8a606803/select.png" alt="" /></p>
<p>在 Discover 页测试效果:</p>
<p><img src="https://www.elastic.co/assets/bltbd8a84ea59294648/highlight-error.png" alt="" /></p>
<h2 id="section-1">更通用化</h2>
<p>插件已经可以运行了,但是我们如果想更通用化一点,不单单可以用来高亮 error 字眼呢?当然不用给每个单词开发一种 formatter,我们可以提供一个输入正则表达式的方式。</p>
<p>在 editor 目录,添加一个叫 <code class="highlighter-rouge">highlight.html</code> 的文件,内容如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o"><</span><span class="nx">div</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"form-group"</span><span class="o">></span>
<span class="o"><</span><span class="nx">label</span><span class="o">></span><span class="nx">Pattern</span><span class="o"><</span><span class="sr">/label</span><span class="err">>
</span> <span class="o"><</span><span class="nx">input</span> <span class="kr">class</span><span class="o">=</span><span class="s2">"form-control"</span> <span class="nx">ng</span><span class="o">-</span><span class="nx">model</span><span class="o">=</span><span class="s2">"editor.formatParams.pattern"</span><span class="o">/></span>
<span class="o"><</span><span class="sr">/div</span><span class="err">>
</span></code></pre>
</div>
<p>然后回到 Highlight.js 里,我们需要定义 <code class="highlighter-rouge">highlight.html</code> 作为我们的编辑页面,然后更新我们的 <code class="highlighter-rouge">_highlight</code> 方法,使用输入文本作为匹配时的正则表达式。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nx">Highlight</span><span class="p">.</span><span class="nx">editor</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'ui/stringify/editors/highlight.html'</span><span class="p">);</span>
<span class="nx">Highlight</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">_highlight</span> <span class="o">=</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">val</span><span class="p">,</span> <span class="nx">replace</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">escapedVal</span> <span class="o">=</span> <span class="nx">_</span><span class="p">.</span><span class="nx">escape</span><span class="p">(</span><span class="nx">val</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">highlightPattern</span><span class="p">;</span>
<span class="k">try</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">inputRegex</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">param</span><span class="p">(</span><span class="s1">'pattern'</span><span class="p">).</span><span class="nx">split</span><span class="p">(</span><span class="s1">'/'</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">pattern</span> <span class="o">=</span> <span class="nx">inputRegex</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">||</span> <span class="nx">inputRegex</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span>
<span class="kd">var</span> <span class="nx">flags</span> <span class="o">=</span> <span class="nx">inputRegex</span><span class="p">[</span><span class="mi">2</span><span class="p">];</span>
<span class="nx">highlightPattern</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">RegExp</span><span class="p">(</span><span class="nx">pattern</span><span class="p">,</span> <span class="nx">flags</span><span class="p">);</span>
<span class="p">}</span> <span class="k">catch</span><span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">escapedVal</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nx">escapedVal</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="nx">highlightPattern</span><span class="p">,</span> <span class="nx">replace</span><span class="p">);</span>
<span class="p">};</span>
</code></pre>
</div>
<h2 id="section-2">示例</h2>
<p>如果在应用 formatter 之前,就能看到输入的正则表达式的效果就更好了。Kibana 里提供了一个 directive 指令让我们可以在修改表达式时观察示例变化。</p>
<p>我们可以增加一些输入字段,并且在模板中加入这个指令。也就是在 highlight.html 后面追加下面这段:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o"><</span><span class="nx">field</span><span class="o">-</span><span class="nx">format</span><span class="o">-</span><span class="nx">editor</span><span class="o">-</span><span class="nx">samples</span> <span class="nx">inputs</span><span class="o">=</span><span class="s2">"editor.field.format.type.sampleInputs"</span><span class="o">><</span><span class="sr">/field-format-editor-samples</span><span class="err">>
</span></code></pre>
</div>
<p>对应的,在 Highlight.js 里添加下面这段:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nx">Highlight</span><span class="p">.</span><span class="nx">sampleInputs</span> <span class="o">=</span> <span class="p">[</span>
<span class="s1">'Hello world'</span><span class="p">,</span>
<span class="s1">'The quick brown fox jumps over the lazy dog'</span><span class="p">,</span>
<span class="s1">'112345'</span>
<span class="p">];</span>
</code></pre>
</div>
<p>最终结果如下:</p>
<p><img src="https://www.elastic.co/assets/blt8bbd181d804191a0/sample.png" alt="" /></p>
<h2 id="section-3">结论</h2>
<p>field formatter 接口提供了非常简便的办法让我们定制字段内容的展示方式。Kibana 自带了好几种 formatter,不过如果你没发现比较合适的,你可以随时自己开发添加一个。如果你已经开始计划添加了,也请注意在 Kibana 4.2 发版的时候,回来看看,有没有新的接口变更。</p>
Elasticsearch 同一索引不同类型下同名字段的映射冲突实例
2015-04-03T00:00:00+08:00
logstash
elasticsearch
http://chenlinux.com/2015/04/03/types-mapping-conflict-in-one-index
<p>这个标题肯定绕晕很多人吧。具体说明一下场景就明白了:Nginx 和 Apache 的访问日志,因为都属于网站访问,所以写入到同一个索引的不同类型下,比方 <code class="highlighter-rouge">logstash-accesslog-2015.04.03/nginx</code> 和 <code class="highlighter-rouge">logstash-accesslog-2015.04.03/apache</code>。既然都是访问日志,肯定很多字段的内容含义是雷同的,比如 clientip, domain, urlpath 等等。其中 nginx 有一个变量叫 <code class="highlighter-rouge">$request_time</code>,apache 有一个变量叫 <code class="highlighter-rouge">%T</code>,乍看上去也是同义的,我就统一命名为 “requestTime” 了。这就是”同一索引(logstash-accesslog-YYYY.MM.DD)下不同类型(nginx,apache)的同名字段(requestTime)”。</p>
<p>但事实上,这里有个问题:<strong>nginx 中的以秒为单位,是把毫秒算作小数;apache 中的以秒为单位,是真的只记秒钟整数位!</strong></p>
<p>所以,这两个类型生成的映射在这个字段上是不一致的。nginx 类型的 requestTime 是 <strong>double</strong>,apache 类型的 requestTime 是 <strong>long</strong>。</p>
<p>不过平常看起来似乎也没什么影响,写入数据都照常,查看数据的时候默认显示的 JSON 也各自无异。直到我准备用一把 scripted field 的时候,发现计算 <code class="highlighter-rouge">doc['requestTime'].value * 1000</code> 得到的数都大的吓人!</p>
<p>因为类似计算之前在只有 nginx 日志入库的时候曾经正确运行过,所以只能是猜测 apache 日志对此造成了影响,但是即使我把请求修改成限定在 nginx 类型数据中进行,结果也没发生变化。</p>
<p>仔细阅读 scripting module 的文档,其中提到了 <code class="highlighter-rouge">doc['fieldname'].value</code> 和 <code class="highlighter-rouge">_source.fieldname</code> 两种写法的区别:<strong>前者会利用内存中的数据,而后者强制读取磁盘上 <code class="highlighter-rouge">_source</code> 存储的 JSON 内容,从中释放出相应字段内容。</strong>莫非是 requestTime 字段跟 <code class="highlighter-rouge">_source</code> JSON 里存的数据确实不一样,而我们平常搜索查看的都是从 JSON 里释放出来的,所以才会如此?</p>
<p>为了验证我的猜测,做了一个请求测试:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># curl es.domain.com:9200/logstash-accesslog-2015.04.03/nginx/_search?q=_id:AUx-QvSBS-dhpiB8_1f1\&pretty -d '{</span>
<span class="s2">"fields"</span>: <span class="o">[</span><span class="s2">"requestTime"</span>, <span class="s2">"bodySent"</span><span class="o">]</span>,
<span class="s2">"script_fields"</span> : <span class="o">{</span>
<span class="s2">"test1"</span> : <span class="o">{</span>
<span class="s2">"script"</span> : <span class="s2">"doc[</span><span class="se">\"</span><span class="s2">requestTime</span><span class="se">\"</span><span class="s2">].value"</span>
<span class="o">}</span>,
<span class="s2">"test3"</span> : <span class="o">{</span>
<span class="s2">"script"</span> : <span class="s2">"_source.bodySent / _source.requestTime"</span>
<span class="o">}</span>,
<span class="s2">"test2"</span> : <span class="o">{</span>
<span class="s2">"script"</span> : <span class="s2">"doc[</span><span class="se">\"</span><span class="s2">requestTime</span><span class="se">\"</span><span class="s2">].value * 1000"</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span><span class="s1">'
</span></code></pre>
</div>
<p>得到的结果如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nt">"took"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">43</span><span class="p">,</span><span class="w">
</span><span class="nt">"timed_out"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nt">"_shards"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">56</span><span class="p">,</span><span class="w">
</span><span class="nt">"successful"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">56</span><span class="p">,</span><span class="w">
</span><span class="nt">"failed"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w">
</span><span class="nt">"max_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-accesslog-2015.04.03"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"nginx"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AUx-QvSBS-dhpiB8_1f1"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">1.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"fields"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"test1"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mi">4603039107142836552</span><span class="w"> </span><span class="p">],</span><span class="w">
</span><span class="nt">"test2"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mi">-8646911284551352000</span><span class="w"> </span><span class="p">],</span><span class="w">
</span><span class="nt">"requestTime"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mf">0.54</span><span class="w"> </span><span class="p">],</span><span class="w">
</span><span class="nt">"test3"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mf">2444.4444444444443</span><span class="w"> </span><span class="p">],</span><span class="w">
</span><span class="nt">"bodySent"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mi">1320</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
</div>
<p>果然!直接读取的字段,以及采用 <code class="highlighter-rouge">_source.fieldname</code> 方式读取的内容,都是正确的;而采用 <code class="highlighter-rouge">doc['fieldname'].value</code> 获取的内存数据,就不对。(0.54 存成 long 型会变成 4603039107142836552。这个 460 还正好能跟 540 凑成 1000,应该是某种特定存法,不过这里我就没深究了)</p>
<p>再作下一步验证。我们知道,ES 数据的映射是根据第一条数据的类型确定的,之后的数据如何类型跟已经成型的映射不统一,那么写入会失败。现在这个 nginx 和 apache 两个类型在 requestTime 字段上的映射是不一样的,但是内存里却并没有按照映射来处理。那么,我往一个类型下写入另一个类型映射要求的数据,会报错还是会通过呢?</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># curl -XPOST es.domain.com:9200/test/t1/1 -d '{"key":1}'</span>
<span class="o">{</span><span class="s2">"_index"</span>:<span class="s2">"test"</span>,<span class="s2">"_type"</span>:<span class="s2">"t1"</span>,<span class="s2">"_id"</span>:<span class="s2">"1"</span>,<span class="s2">"_version"</span>:1,<span class="s2">"created"</span>:true<span class="o">}</span>
<span class="c"># curl -XPOST es.domain.com:9200/test/t2/1 -d '{"key":2.2}'</span>
<span class="o">{</span><span class="s2">"_index"</span>:<span class="s2">"test"</span>,<span class="s2">"_type"</span>:<span class="s2">"t2"</span>,<span class="s2">"_id"</span>:<span class="s2">"1"</span>,<span class="s2">"_version"</span>:1,<span class="s2">"created"</span>:true<span class="o">}</span>
<span class="c"># curl -XPOST es.domain.com:9200/test/t1/2 -d '{"key":2.2}'</span>
<span class="o">{</span><span class="s2">"_index"</span>:<span class="s2">"test"</span>,<span class="s2">"_type"</span>:<span class="s2">"t1"</span>,<span class="s2">"_id"</span>:<span class="s2">"2"</span>,<span class="s2">"_version"</span>:1,<span class="s2">"created"</span>:true<span class="o">}</span>
<span class="c"># curl -XPOST es.domain.com:9200/test/t2/2 -d '{"key":1}'</span>
<span class="o">{</span><span class="s2">"_index"</span>:<span class="s2">"test"</span>,<span class="s2">"_type"</span>:<span class="s2">"t2"</span>,<span class="s2">"_id"</span>:<span class="s2">"2"</span>,<span class="s2">"_version"</span>:1,<span class="s2">"created"</span>:true<span class="o">}</span>
<span class="c"># curl -XPOST es.domain.com:9200/test/t1/3 -d '{"key":"1"}'</span>
<span class="o">{</span><span class="s2">"_index"</span>:<span class="s2">"test"</span>,<span class="s2">"_type"</span>:<span class="s2">"t1"</span>,<span class="s2">"_id"</span>:<span class="s2">"3"</span>,<span class="s2">"_version"</span>:1,<span class="s2">"created"</span>:true<span class="o">}</span>
<span class="c"># curl -XPOST es.domain.com:9200/test/t2/3 -d '{"key":"1"}'</span>
<span class="o">{</span><span class="s2">"_index"</span>:<span class="s2">"test"</span>,<span class="s2">"_type"</span>:<span class="s2">"t2"</span>,<span class="s2">"_id"</span>:<span class="s2">"3"</span>,<span class="s2">"_version"</span>:1,<span class="s2">"created"</span>:true<span class="o">}</span>
<span class="c"># curl -XPOST es.domain.com:9200/test/t2/4 -d '{"key":"abc"}'</span>
<span class="o">{</span><span class="s2">"error"</span>:<span class="s2">"RemoteTransportException[[10.10.10.10][inet[/10.10.10.10:9300]][indices:data/write/index]]; nested: MapperParsingException[failed to parse [key]]; nested: NumberFormatException[For input string: </span><span class="se">\"</span><span class="s2">abc</span><span class="se">\"</span><span class="s2">]; "</span>,<span class="s2">"status"</span>:400<span class="o">}</span>
<span class="c"># curl -XGET es.domain.com:9200/test/_mapping</span>
<span class="o">{</span><span class="s2">"test"</span>:<span class="o">{</span><span class="s2">"mappings"</span>:<span class="o">{</span><span class="s2">"t1"</span>:<span class="o">{</span><span class="s2">"properties"</span>:<span class="o">{</span><span class="s2">"key"</span>:<span class="o">{</span><span class="s2">"type"</span>:<span class="s2">"long"</span><span class="o">}}}</span>,<span class="s2">"t2"</span>:<span class="o">{</span><span class="s2">"properties"</span>:<span class="o">{</span><span class="s2">"key"</span>:<span class="o">{</span><span class="s2">"type"</span>:<span class="s2">"double"</span><span class="o">}}}}}}</span>
</code></pre>
</div>
<p>结果出来了,在映射相互冲突以后,实际数据只要是 numeric detect 能通过的,就都通过了!</p>
<p>BTW: kibana 4 中,已经会对这种情况以黄色感叹号图标做出提示;而根据官方消息,ES 未来会在 2.0 版正式杜绝这种可能。</p>
spark streaming 接收 kafka 数据示例
2015-03-14T00:00:00+08:00
monitor
spark
kafka
scala
http://chenlinux.com/2015/03/14/spark-streaming-kafka
<p>上个月曾经试过了用 spark streaming 读取 logstash 启动的 TCP Server 的数据。不过如果你有多台 logstash 的时候,这种方式就比较难办了 —— 即使你给 logstash 集群申请一个 VIP,也很难确定说转发完全符合。所以一般来说,更多的选择是采用 kafka 等队列方式由 spark streaming 去作为订阅者获取数据。</p>
<h2 id="section">环境部署</h2>
<p>这里只讲 kafka 单机的部署。只是示例嘛:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>cd kafka_2.10-0.8.2.0/bin/
./zookeeper-server-start.sh ../config/zookeeper.properties &
./kafka-server-start.sh --daemon ../config/server.properties
</code></pre>
</div>
<h2 id="section-1">数据转发</h2>
<p>保持跟之前示例的连贯性,这里继续用 logstash 发送数据到 kafka。</p>
<p>首先创建一个 kafka 的 topic:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic logstash
</code></pre>
</div>
<p>然后到 logstash 里,修改配置为:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>input {
file { path => "/var/log/*.log" }
}
filter {
ruby {
code => "event['lineno'] = 100 * rand(Math::E..Math::PI)"
}
}
output {
kafka {
broker_list => "127.0.0.1:9092"
topic_id => "logstash"
}
}
</code></pre>
</div>
<h2 id="spark-streaming-">spark streaming 处理的代码:</h2>
<p>处理效果跟之前示例依然保持一致,就不重复贴冗余的函数了,只贴最开始的处理部分:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">org.apache.spark.SparkConf</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.SparkContext</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.SparkContext._</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.streaming.</span><span class="o">{</span><span class="n">Seconds</span><span class="o">,</span> <span class="n">StreamingContext</span><span class="o">}</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.streaming.StreamingContext._</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.streaming.kafka.KafkaUtils</span>
<span class="kn">import</span> <span class="nn">org.json4s._</span>
<span class="kn">import</span> <span class="nn">org.json4s.jackson.JsonMethods._</span>
<span class="n">object</span> <span class="n">LogStash</span> <span class="o">{</span>
<span class="n">implicit</span> <span class="n">val</span> <span class="n">formats</span> <span class="o">=</span> <span class="n">DefaultFormats</span>
<span class="k">case</span> <span class="kd">class</span> <span class="nc">LogStashV1</span><span class="o">(</span><span class="nl">message:</span><span class="n">String</span><span class="o">,</span> <span class="nl">path:</span><span class="n">String</span><span class="o">,</span> <span class="nl">host:</span><span class="n">String</span><span class="o">,</span> <span class="nl">lineno:</span><span class="n">Double</span><span class="o">,</span> <span class="err">`</span><span class="nd">@timestamp</span><span class="err">`</span><span class="o">:</span><span class="n">String</span><span class="o">)</span>
<span class="n">def</span> <span class="n">main</span><span class="o">(</span><span class="nl">args:</span> <span class="n">Array</span><span class="o">[</span><span class="n">String</span><span class="o">])</span> <span class="o">{</span>
<span class="n">val</span> <span class="n">Array</span><span class="o">(</span><span class="n">zkQuorum</span><span class="o">,</span> <span class="n">group</span><span class="o">,</span> <span class="n">topics</span><span class="o">,</span> <span class="n">numThreads</span><span class="o">)</span> <span class="o">=</span> <span class="n">args</span>
<span class="n">val</span> <span class="n">topicMap</span> <span class="o">=</span> <span class="n">topics</span><span class="o">.</span><span class="na">split</span><span class="o">(</span><span class="s">","</span><span class="o">).</span><span class="na">map</span><span class="o">((</span><span class="n">_</span><span class="o">,</span><span class="n">numThreads</span><span class="o">.</span><span class="na">toInt</span><span class="o">)).</span><span class="na">toMap</span>
<span class="n">val</span> <span class="n">sparkConf</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SparkConf</span><span class="o">().</span><span class="na">setMaster</span><span class="o">(</span><span class="s">"local[2]"</span><span class="o">).</span><span class="na">setAppName</span><span class="o">(</span><span class="s">"LogStash"</span><span class="o">)</span>
<span class="n">val</span> <span class="n">sc</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SparkContext</span><span class="o">(</span><span class="n">sparkConf</span><span class="o">)</span>
<span class="n">val</span> <span class="n">ssc</span> <span class="o">=</span> <span class="k">new</span> <span class="n">StreamingContext</span><span class="o">(</span><span class="n">sc</span><span class="o">,</span> <span class="n">Seconds</span><span class="o">(</span><span class="mi">10</span><span class="o">))</span>
<span class="n">val</span> <span class="n">lines</span> <span class="o">=</span> <span class="n">KafkaUtils</span><span class="o">.</span><span class="na">createStream</span><span class="o">(</span><span class="n">ssc</span><span class="o">,</span> <span class="n">zkQuorum</span><span class="o">,</span> <span class="n">group</span><span class="o">,</span> <span class="n">topicMap</span><span class="o">).</span><span class="na">map</span><span class="o">(</span><span class="n">_</span><span class="o">.</span><span class="na">_2</span><span class="o">)</span>
<span class="n">lines</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">line</span> <span class="o">=></span> <span class="o">{</span>
<span class="n">val</span> <span class="n">json</span> <span class="o">=</span> <span class="n">parse</span><span class="o">(</span><span class="n">line</span><span class="o">)</span>
<span class="n">json</span><span class="o">.</span><span class="na">extract</span><span class="o">[</span><span class="n">LogStashV1</span><span class="o">]</span>
<span class="o">}).</span><span class="na">print</span><span class="o">()</span>
<span class="n">ssc</span><span class="o">.</span><span class="na">start</span><span class="o">()</span>
<span class="n">ssc</span><span class="o">.</span><span class="na">awaitTermination</span><span class="o">()</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre>
</div>
<p>这里面有一些跟网上常见资料不一样的地方。</p>
<p>第一个,<code class="highlighter-rouge">import org.apache.spark.streaming.kafka._</code> 并不会导出 <code class="highlighter-rouge">KafkaUtils</code>,必须明确写明才行。<br />
第二个,之前示例里用了 scala 核心自带的 JSON 模块。但是这次我把 lineno 字段从整数改成浮点数后,发现 <code class="highlighter-rouge">JSON.parseFull()</code> 有问题。虽然我在 scala 的 repl 里测试没问题,但是写在 spark 里的时候,它并不像文档所说的”总是尝试解析成 Double 类型”,而是一直尝试用 <code class="highlighter-rouge">Integer.parseInteger()</code> 方法来解析。哪怕我明确定义 <code class="highlighter-rouge">JSON.globalNumberParser = {input:String => Float.parseFloat(input)}</code> 都不起作用。</p>
<p>所以,最后这里改用了 <a href="http://json4s.org">json4s 库</a>。据称这也是 scala 里性能和功能最好的 JSON 库。</p>
<p>json4s 库默认解析完后,不是标准的 Map、List 等对象,而是它自己的 JObject、JList、JString 等。想要转换成标准 scala 对象,需要调用 <code class="highlighter-rouge">.values</code> 才对。不过我这个示例里没有这么麻烦,而是直接采用 <code class="highlighter-rouge">.extract</code> 就变成了 cast class 对象了。非常简便。</p>
<p>另一个需要点出来的变动是:因为采用 <code class="highlighter-rouge">.extract</code>,所以 cast class 里的参数命名必须跟 JSON 里的 key 完全对应上。而我们都知道 logstash 里有几个特殊的字段,叫 <code class="highlighter-rouge">@timestamp</code> 和 <code class="highlighter-rouge">@version</code> 。这个 “@” 是不能直接裸字符的,所以要用反引号(<strong>`</strong>)包括起来。</p>
<h2 id="sbt-">sbt 打包</h2>
<p>sbt 打包也需要有所变动。spark streaming 的核心代码中,并不包含 kafka 的代码。还跟之前那样 <code class="highlighter-rouge">sbt package</code> 的话,就得另外指定 kafka 的 jar 地址才能运行了。更合适的办法,是打包一个完全包含的 jar 包。这就用到 <a href="https://github.com/sbt/sbt-assembly">sbt-assembly 扩展</a>。</p>
<p><em>刚刚收到的消息,spark 1.3 版发布 beta 了,spark streaming 会内置对 kafka 的底层直接支持。或许以后不用这么麻烦?</em></p>
<p>sbt-assembly 使用起来特别简单,尤其是当你使用的 sbt 版本比较新(大于 0.13.6) 的时候。</p>
<ol>
<li>添加扩展</li>
</ol>
<p>在项目的 <code class="highlighter-rouge">project/</code> 目录下创建一个 <code class="highlighter-rouge">plugins.sbt</code> 文件,内容如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")
</code></pre>
</div>
<p>具体的版本选择,看官方 README 的 <a href="https://github.com/sbt/sbt-assembly#setup">Setup 部分</a>。</p>
<ol>
<li>添加新增依赖模块</li>
</ol>
<p>现在可以去修改我们项目的 <code class="highlighter-rouge">build.sbt</code> 了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">name</span> <span class="o">:=</span> <span class="s">"LogStash"</span>
<span class="n">version</span> <span class="o">:=</span> <span class="s">"1.0"</span>
<span class="n">scalaVersion</span> <span class="o">:=</span> <span class="s">"2.10.4"</span>
<span class="n">libraryDependencies</span> <span class="o">++=</span> <span class="nc">Seq</span><span class="o">(</span>
<span class="s">"org.apache.spark"</span> <span class="o">%%</span> <span class="s">"spark-core"</span> <span class="o">%</span> <span class="s">"1.2.0"</span> <span class="o">%</span> <span class="s">"provided"</span><span class="o">,</span>
<span class="s">"org.apache.spark"</span> <span class="o">%%</span> <span class="s">"spark-sql"</span> <span class="o">%</span> <span class="s">"1.2.0"</span> <span class="o">%</span> <span class="s">"provided"</span><span class="o">,</span>
<span class="s">"org.apache.spark"</span> <span class="o">%%</span> <span class="s">"spark-streaming"</span> <span class="o">%</span> <span class="s">"1.2.0"</span> <span class="o">%</span> <span class="s">"provided"</span><span class="o">,</span>
<span class="s">"org.apache.spark"</span> <span class="o">%%</span> <span class="s">"spark-streaming-kafka"</span> <span class="o">%</span> <span class="s">"1.2.0"</span><span class="o">,</span>
<span class="s">"org.json4s"</span> <span class="o">%%</span> <span class="s">"json4s-native"</span> <span class="o">%</span> <span class="s">"3.2.10"</span><span class="o">,</span>
<span class="s">"org.json4s"</span> <span class="o">%%</span> <span class="s">"json4s-jackson"</span> <span class="o">%</span> <span class="s">"3.2.10"</span>
<span class="o">)</span>
</code></pre>
</div>
<p>是的。新版本的 sbt-assembly 完全不需要单独修改 <code class="highlighter-rouge">build.sbt</code> 了。</p>
<p>需要注意,因为我们这次是需要把各种依赖全部打包到一起,这个可能会导致一些文件相互有冲突。比如我们用 spark-submit 提交任务,有关 spark 的核心文件,本身里面就已经有了的,那么就需要额外通过 <code class="highlighter-rouge">% "provided"</code> 指明这部分会另外提供,不需要打进去。这样运行的时候就不会有问题了。</p>
<ol>
<li>打包</li>
</ol>
<p>采用 sbt-assembly 后的打包命令是:<code class="highlighter-rouge">sbt assembly</code>。注意输出的结果,会是直接读取 <code class="highlighter-rouge">build.sbt</code> 里的 <code class="highlighter-rouge">name</code> 变量,不做处理。,我们之前定义的叫 “LogStash Project”,<code class="highlighter-rouge">sbt package</code> 命令自动会转换成全小写且空格改成中横线的格式 <em>logstash-project_2.10-1.0.jar</em>。但是 <code class="highlighter-rouge">sbt assembly</code> 就会打包成 <em>LogStash Project-assembly-1.0.jar</em> 包。这个空格在走 spark-submit 提交的时候是有问题的。所以这里需要把 name 改成一个不会中断的字符串。。。</p>
Kibana 3 源码解析
2015-03-14T00:00:00+08:00
logstash
kibana
angularjs
http://chenlinux.com/2015/03/14/kibana3-source-code-analysis
<p>本文之前已经拆分成章节发布在我的 <a href="http://kibana.logstash.es/">《Kibana 权威指南》电子书</a>上。欢迎移步观看全书其他章节。</p>
<hr />
<p>Kibana 3 作为 ELKstack 风靡世界的最大推动力,其与优美的界面配套的简洁的代码同样功不可没。事实上,graphite 社区就通过移植 kibana 3 代码框架的方式,启动了 <a href="http://grafana.org/">grafana 项目</a>。至今你还能在 grafana 源码找到二十多处 “kbn” 字样。</p>
<p><em>巧合的是,在 Kibana 重构 v4 版的同时,grafana 的 v2 版也到了 Alpha 阶段,从目前的预览效果看,主体 dashboard 沿用了 Kibana 3 的风格,不过添加了额外的菜单栏,供用户权限设置等使用 —— 这意味着 grafana 2 跟 kibana 4 一样需要一个单独的 server 端。</em></p>
<p>笔者并非专业的前端工程师,对 angularjs 也处于一本入门指南都没看过的水准。所以本节内容,只会抽取一些个人经验中会有涉及到的地方提出一些”私货”。欢迎方家指正。</p>
<h2 id="section">源码目录结构</h2>
<p>下面是 kibana 源码的全部文件的 tree 图:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>.
├── app
│ ├── app.js
│ ├── components
│ │ ├── extend-jquery.js
│ │ ├── kbn.js
│ │ ├── lodash.extended.js
│ │ ├── require.config.js
│ │ └── settings.js
│ ├── controllers
│ │ ├── all.js
│ │ ├── dash.js
│ │ ├── dashLoader.js
│ │ ├── pulldown.js
│ │ └── row.js
│ ├── dashboards
│ │ ├── blank.json
│ │ ├── default.json
│ │ ├── guided.json
│ │ ├── logstash.js
│ │ ├── logstash.json
│ │ ├── noted.json
│ │ ├── panel.js
│ │ └── test.json
│ ├── directives
│ │ ├── addPanel.js
│ │ ├── all.js
│ │ ├── arrayJoin.js
│ │ ├── configModal.js
│ │ ├── confirmClick.js
│ │ ├── dashUpload.js
│ │ ├── esVersion.js
│ │ ├── kibanaPanel.js
│ │ ├── kibanaSimplePanel.js
│ │ ├── ngBlur.js
│ │ ├── ngModelOnBlur.js
│ │ ├── resizable.js
│ │ └── tip.js
│ ├── factories
│ │ └── store.js
│ ├── filters
│ │ └── all.js
│ ├── panels
│ │ ├── bettermap
│ │ │ ├── editor.html
│ │ │ ├── leaflet
│ │ │ │ ├── images
│ │ │ │ │ ├── layers-2x.png
│ │ │ │ │ ├── layers.png
│ │ │ │ │ ├── marker-icon-2x.png
│ │ │ │ │ ├── marker-icon.png
│ │ │ │ │ └── marker-shadow.png
│ │ │ │ ├── leaflet-src.js
│ │ │ │ ├── leaflet.css
│ │ │ │ ├── leaflet.ie.css
│ │ │ │ ├── leaflet.js
│ │ │ │ ├── plugins.css
│ │ │ │ ├── plugins.js
│ │ │ │ └── providers.js
│ │ │ ├── module.css
│ │ │ ├── module.html
│ │ │ └── module.js
│ │ ├── column
│ │ │ ├── editor.html
│ │ │ ├── module.html
│ │ │ ├── module.js
│ │ │ └── panelgeneral.html
│ │ ├── dashcontrol
│ │ │ ├── editor.html
│ │ │ ├── module.html
│ │ │ └── module.js
│ │ ├── derivequeries
│ │ │ ├── editor.html
│ │ │ ├── module.html
│ │ │ └── module.js
│ │ ├── fields
│ │ │ ├── editor.html
│ │ │ ├── micropanel.html
│ │ │ ├── module.html
│ │ │ └── module.js
│ │ ├── filtering
│ │ │ ├── editor.html
│ │ │ ├── meta.html
│ │ │ ├── module.html
│ │ │ └── module.js
│ │ ├── force
│ │ │ ├── editor.html
│ │ │ ├── module.html
│ │ │ └── module.js
│ │ ├── goal
│ │ │ ├── editor.html
│ │ │ ├── module.html
│ │ │ └── module.js
│ │ ├── histogram
│ │ │ ├── editor.html
│ │ │ ├── interval.js
│ │ │ ├── module.html
│ │ │ ├── module.js
│ │ │ ├── queriesEditor.html
│ │ │ ├── styleEditor.html
│ │ │ └── timeSeries.js
│ │ ├── hits
│ │ │ ├── editor.html
│ │ │ ├── module.html
│ │ │ └── module.js
│ │ ├── map
│ │ │ ├── editor.html
│ │ │ ├── lib
│ │ │ │ ├── jquery.jvectormap.min.js
│ │ │ │ ├── map.cn.js
│ │ │ │ ├── map.europe.js
│ │ │ │ ├── map.usa.js
│ │ │ │ └── map.world.js
│ │ │ ├── module.html
│ │ │ └── module.js
│ │ ├── multifieldhistogram
│ │ │ ├── editor.html
│ │ │ ├── interval.js
│ │ │ ├── markersEditor.html
│ │ │ ├── meta.html
│ │ │ ├── module.html
│ │ │ ├── module.js
│ │ │ ├── styleEditor.html
│ │ │ └── timeSeries.js
│ │ ├── percentiles
│ │ │ ├── editor.html
│ │ │ ├── module.html
│ │ │ └── module.js
│ │ ├── query
│ │ │ ├── editor.html
│ │ │ ├── editors
│ │ │ │ ├── lucene.html
│ │ │ │ ├── regex.html
│ │ │ │ └── topN.html
│ │ │ ├── help
│ │ │ │ ├── lucene.html
│ │ │ │ ├── regex.html
│ │ │ │ └── topN.html
│ │ │ ├── helpModal.html
│ │ │ ├── meta.html
│ │ │ ├── module.html
│ │ │ ├── module.js
│ │ │ └── query.css
│ │ ├── ranges
│ │ │ ├── editor.html
│ │ │ ├── module.html
│ │ │ └── module.js
│ │ ├── sparklines
│ │ │ ├── editor.html
│ │ │ ├── interval.js
│ │ │ ├── module.html
│ │ │ ├── module.js
│ │ │ └── timeSeries.js
│ │ ├── statisticstrend
│ │ │ ├── editor.html
│ │ │ ├── module.html
│ │ │ └── module.js
│ │ ├── stats
│ │ │ ├── editor.html
│ │ │ ├── module.html
│ │ │ └── module.js
│ │ ├── table
│ │ │ ├── editor.html
│ │ │ ├── export.html
│ │ │ ├── micropanel.html
│ │ │ ├── modal.html
│ │ │ ├── module.html
│ │ │ ├── module.js
│ │ │ └── pagination.html
│ │ ├── terms
│ │ │ ├── editor.html
│ │ │ ├── module.html
│ │ │ └── module.js
│ │ ├── text
│ │ │ ├── editor.html
│ │ │ ├── lib
│ │ │ │ └── showdown.js
│ │ │ ├── module.html
│ │ │ └── module.js
│ │ ├── timepicker
│ │ │ ├── custom.html
│ │ │ ├── editor.html
│ │ │ ├── module.html
│ │ │ ├── module.js
│ │ │ └── refreshctrl.html
│ │ ├── trends
│ │ │ ├── editor.html
│ │ │ ├── module.html
│ │ │ └── module.js
│ │ └── valuehistogram
│ │ ├── editor.html
│ │ ├── module.html
│ │ ├── module.js
│ │ ├── queriesEditor.html
│ │ └── styleEditor.html
│ ├── partials
│ │ ├── connectionFailed.html
│ │ ├── dashLoader.html
│ │ ├── dashLoaderShare.html
│ │ ├── dashboard.html
│ │ ├── dasheditor.html
│ │ ├── inspector.html
│ │ ├── load.html
│ │ ├── modal.html
│ │ ├── paneladd.html
│ │ ├── paneleditor.html
│ │ ├── panelgeneral.html
│ │ ├── querySelect.html
│ │ └── roweditor.html
│ └── services
│ ├── alertSrv.js
│ ├── all.js
│ ├── dashboard.js
│ ├── esVersion.js
│ ├── fields.js
│ ├── filterSrv.js
│ ├── kbnIndex.js
│ ├── monitor.js
│ ├── panelMove.js
│ ├── querySrv.js
│ └── timer.js
├── config.js
├── css
│ ├── angular-multi-select.css
│ ├── animate.min.css
│ ├── bootstrap-responsive.min.css
│ ├── bootstrap.dark.min.css
│ ├── bootstrap.light.min.css
│ ├── font-awesome.min.css
│ ├── jquery-ui.css
│ ├── jquery.multiselect.css
│ ├── normalize.min.css
│ └── timepicker.css
├── favicon.ico
├── font
│ ├── FontAwesome.otf
│ ├── fontawesome-webfont.eot
│ ├── fontawesome-webfont.svg
│ ├── fontawesome-webfont.ttf
│ └── fontawesome-webfont.woff
├── img
│ ├── annotation-icon.png
│ ├── cubes.png
│ ├── glyphicons-halflings-white.png
│ ├── glyphicons-halflings.png
│ ├── kibana.png
│ ├── light.png
│ ├── load.gif
│ ├── load_big.gif
│ ├── small.png
│ └── ui-icons_222222_256x240.png
├── index.html
└── vendor
├── LICENSE.json
├── angular
│ ├── angular-animate.js
│ ├── angular-cookies.js
│ ├── angular-dragdrop.js
│ ├── angular-loader.js
│ ├── angular-resource.js
│ ├── angular-route.js
│ ├── angular-sanitize.js
│ ├── angular-scenario.js
│ ├── angular-strap.js
│ ├── angular.js
│ ├── bindonce.js
│ ├── datepicker.js
│ └── timepicker.js
├── blob.js
├── bootstrap
│ ├── bootstrap.js
│ └── less
│ ├── accordion.less
│ ├── alerts.less
│ ├── bak
│ │ ├── bootswatch.dark.less
│ │ └── variables.dark.less
│ ├── bootstrap.dark.less
│ ├── bootstrap.less
│ ├── bootstrap.light.less
│ ├── bootswatch.dark.less
│ ├── bootswatch.light.less
│ ├── breadcrumbs.less
│ ├── button-groups.less
│ ├── buttons.less
│ ├── carousel.less
│ ├── close.less
│ ├── code.less
│ ├── component-animations.less
│ ├── dropdowns.less
│ ├── forms.less
│ ├── grid.less
│ ├── hero-unit.less
│ ├── labels-badges.less
│ ├── layouts.less
│ ├── media.less
│ ├── mixins.less
│ ├── modals.less
│ ├── navbar.less
│ ├── navs.less
│ ├── overrides.less
│ ├── pager.less
│ ├── pagination.less
│ ├── popovers.less
│ ├── progress-bars.less
│ ├── reset.less
│ ├── responsive-1200px-min.less
│ ├── responsive-767px-max.less
│ ├── responsive-768px-979px.less
│ ├── responsive-navbar.less
│ ├── responsive-utilities.less
│ ├── responsive.less
│ ├── scaffolding.less
│ ├── sprites.less
│ ├── tables.less
│ ├── tests
│ │ ├── buttons.html
│ │ ├── css-tests.css
│ │ ├── css-tests.html
│ │ ├── forms-responsive.html
│ │ ├── forms.html
│ │ ├── navbar-fixed-top.html
│ │ ├── navbar-static-top.html
│ │ └── navbar.html
│ ├── thumbnails.less
│ ├── tooltip.less
│ ├── type.less
│ ├── utilities.less
│ ├── variables.dark.less
│ ├── variables.less
│ ├── variables.light.less
│ └── wells.less
├── chromath.js
├── elasticjs
│ ├── elastic-angular-client.js
│ └── elastic.js
├── elasticsearch.angular.js
├── filesaver.js
├── jquery
│ ├── jquery-1.8.0.js
│ ├── jquery-ui-1.10.3.js
│ ├── jquery.flot.byte.js
│ ├── jquery.flot.events.js
│ ├── jquery.flot.js
│ ├── jquery.flot.pie.js
│ ├── jquery.flot.selection.js
│ ├── jquery.flot.stack.js
│ ├── jquery.flot.stackpercent.js
│ ├── jquery.flot.threshold.js
│ ├── jquery.flot.time.js
│ ├── jquery.multiselect.filter.js
│ └── jquery.multiselect.js
├── jsonpath.js
├── lodash.js
├── modernizr-2.6.1.js
├── moment.js
├── numeral.js
├── require
│ ├── css-build.js
│ ├── css.js
│ ├── require.js
│ ├── text.js
│ └── tmpl.js
├── simple_statistics.js
├── timezone.js
└── underscore.string.js
</code></pre>
</div>
<p>一目了然,我们可以归纳出下面几类主要文件:</p>
<ul>
<li>入口:index.html</li>
<li>模块库:vendor/</li>
<li>程序入口:app/app.js</li>
<li>组件配置:app/components/</li>
<li>仪表板控制:app/controllers/</li>
<li>挂件页面:app/partials/</li>
<li>服务:app/services/</li>
<li>指令:app/directives/</li>
<li>图表:app/panels/</li>
</ul>
<h2 id="section-1">入口和模块依赖</h2>
<p>这一部分是网页项目的基础。从 index.html 里就可以学到 angularjs 最基础的常用模板语法了。出现的指令有:<code class="highlighter-rouge">ng-repeat</code>, <code class="highlighter-rouge">ng-controller</code>, <code class="highlighter-rouge">ng-include</code>, <code class="highlighter-rouge">ng-view</code>, <code class="highlighter-rouge">ng-slow</code>, <code class="highlighter-rouge">ng-click</code>, <code class="highlighter-rouge">ng-href</code>,以及变量绑定的语法:``。</p>
<p>index.html 中,需要注意 js 的加载次序,先 <code class="highlighter-rouge">require.js</code>,然后再 <code class="highlighter-rouge">require.config.js</code>,最后 <code class="highlighter-rouge">app</code>。整个 kibana 项目都是通过 <strong>requrie</strong> 方式加载的。而具体的模块,和模块的依赖关系,则定义在 <code class="highlighter-rouge">require.config.js</code> 里。这些全部加载完成后,才是启动 app 模块,也就是项目本身的代码。</p>
<p>require.config.js 中,主要分成两部分配置,一个是 <code class="highlighter-rouge">paths</code>,一个是 <code class="highlighter-rouge">shim</code>。paths 用来指定依赖模块的导出名称和模块 js 文件的具体路径。而 shim 用来指定依赖模块之间的依赖关系。比方说:绘制图表的 js,kibana3 里用的是 jquery.flot 库。这个就首先依赖于 jquery 库。(通俗的说,就是原先普通的 HTML 写法里,要先加载 jquery.js 再加载 jquery.flot.js)</p>
<p>在整个 paths 中,需要单独提一下的是 <code class="highlighter-rouge">elasticjs:'../vendor/elasticjs/elastic-angular-client'</code>。这是串联 elastic.js 和 angular.js 的文件。这里面实际是定义了一个 angular.module 的 factory,名叫 <code class="highlighter-rouge">ejsResource</code>。后续我们在 kibana 3 里用到的跟 Elasticsearch 交互的所有方法,都在这个 <code class="highlighter-rouge">ejsResource</code> 里了。</p>
<p><em>factory 是 angular 的一个单例对象,创建之后会持续到你关闭浏览器。Kibana 3 就是通过这种方式来控制你所有的图表是从同一个 Elasticsearch 获取的数据</em></p>
<p>app.js 中,定义了整个应用的 routes,加载了 controller, directives 和 filters 里的全部内容。就是在这里,加载了主页面 <code class="highlighter-rouge">app/partials/dashboard.html</code>。当然,这个页面其实没啥看头,因为里面就是提供 pulldown 和 row 的 div,然后绑定到对应的 controller 上。</p>
<h2 id="controller--service">controller 和 service</h2>
<p>controller 里没太多可讲的。kibana 3 里,pulldown 其实跟 row 差别不大,看这简单的几行代码里,最关键的就是几个注入:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nx">define</span><span class="p">([</span><span class="s1">'angular'</span><span class="p">,</span><span class="s1">'app'</span><span class="p">,</span><span class="s1">'lodash'</span><span class="p">],</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">angular</span><span class="p">,</span> <span class="nx">app</span><span class="p">,</span> <span class="nx">_</span><span class="p">)</span> <span class="p">{</span>
<span class="s1">'use strict'</span><span class="p">;</span>
<span class="nx">angular</span><span class="p">.</span><span class="nx">module</span><span class="p">(</span><span class="s1">'kibana.controllers'</span><span class="p">).</span><span class="nx">controller</span><span class="p">(</span><span class="s1">'RowCtrl'</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">$scope</span><span class="p">,</span> <span class="nx">$rootScope</span><span class="p">,</span> <span class="nx">$timeout</span><span class="p">,</span><span class="nx">ejsResource</span><span class="p">,</span> <span class="nx">querySrv</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">_d</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">title</span><span class="p">:</span> <span class="s2">"Row"</span><span class="p">,</span>
<span class="na">height</span><span class="p">:</span> <span class="s2">"150px"</span><span class="p">,</span>
<span class="na">collapse</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="na">collapsable</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="na">editable</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="na">panels</span><span class="p">:</span> <span class="p">[],</span>
<span class="na">notice</span><span class="p">:</span> <span class="kc">false</span>
<span class="p">};</span>
<span class="nx">_</span><span class="p">.</span><span class="nx">defaults</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">row</span><span class="p">,</span><span class="nx">_d</span><span class="p">);</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">init</span> <span class="o">=</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">querySrv</span> <span class="o">=</span> <span class="nx">querySrv</span><span class="p">;</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">reset_panel</span><span class="p">();</span>
<span class="p">};</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">init</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">);</span>
<span class="p">});</span>
</code></pre>
</div>
<p>这里面,注入了 <code class="highlighter-rouge">$scope</code>, <code class="highlighter-rouge">ejsResource</code> 和 <code class="highlighter-rouge">querySrv</code>。<code class="highlighter-rouge">$scope</code> 是控制器作用域内的模型数据对象,这是 angular 提供的一个特殊变量。<code class="highlighter-rouge">ejsResource</code> 是一个 factory ,前面已经讲过。<code class="highlighter-rouge">querySrv</code> 是一个 service,下面说一下。</p>
<p>service 跟 factory 的概念非常类似,一般来说,可能 factory 偏向用来共享一个类,而 service 用来共享一组函数功能。</p>
<p>kibana 3 里,比较有用和常用的 services 包括:</p>
<h3 id="dashboard">dashboard</h3>
<p>dashboard.js 里提供了关于 Kibana 3 仪表板的读写操作。其中主要的几个是提供了三种读取仪表板布局纲要的方式,也就是读取文件,读取存在 <code class="highlighter-rouge">.kibana-int</code> 索引里的数据,读取 js 脚本。下面是读取 js 脚本的相关函数:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">this</span><span class="p">.</span><span class="nx">script_load</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">file</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">$http</span><span class="p">({</span>
<span class="na">url</span><span class="p">:</span> <span class="s2">"app/dashboards/"</span><span class="o">+</span><span class="nx">file</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="sr">/</span><span class="se">\.(?!</span><span class="sr">js</span><span class="se">)</span><span class="sr">/</span><span class="p">,</span><span class="s2">"/"</span><span class="p">),</span>
<span class="na">method</span><span class="p">:</span> <span class="s2">"GET"</span><span class="p">,</span>
<span class="na">transformResponse</span><span class="p">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">response</span><span class="p">)</span> <span class="p">{</span>
<span class="cm">/*jshint -W054 */</span>
<span class="kd">var</span> <span class="nx">_f</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Function</span><span class="p">(</span><span class="s1">'ARGS'</span><span class="p">,</span><span class="s1">'kbn'</span><span class="p">,</span><span class="s1">'_'</span><span class="p">,</span><span class="s1">'moment'</span><span class="p">,</span><span class="s1">'window'</span><span class="p">,</span><span class="s1">'document'</span><span class="p">,</span><span class="s1">'angular'</span><span class="p">,</span><span class="s1">'require'</span><span class="p">,</span><span class="s1">'define'</span><span class="p">,</span><span class="s1">'$'</span><span class="p">,</span><span class="s1">'jQuery'</span><span class="p">,</span><span class="nx">response</span><span class="p">);</span>
<span class="k">return</span> <span class="nx">_f</span><span class="p">(</span><span class="nx">$routeParams</span><span class="p">,</span><span class="nx">kbn</span><span class="p">,</span><span class="nx">_</span><span class="p">,</span><span class="nx">moment</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}).</span><span class="nx">then</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">result</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="o">!</span><span class="nx">result</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="kc">false</span><span class="p">;</span>
<span class="p">}</span>
<span class="nx">self</span><span class="p">.</span><span class="nx">dash_load</span><span class="p">(</span><span class="nx">dash_defaults</span><span class="p">(</span><span class="nx">result</span><span class="p">.</span><span class="nx">data</span><span class="p">));</span>
<span class="k">return</span> <span class="kc">true</span><span class="p">;</span>
<span class="p">},</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">alertSrv</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="s1">'Error'</span><span class="p">,</span>
<span class="s2">"Could not load <i>scripts/"</span><span class="o">+</span><span class="nx">file</span><span class="o">+</span><span class="s2">"</i>. Please make sure it exists and returns a valid dashboard"</span> <span class="p">,</span>
<span class="s1">'error'</span><span class="p">);</span>
<span class="k">return</span> <span class="kc">false</span><span class="p">;</span>
<span class="p">});</span>
<span class="p">};</span>
</code></pre>
</div>
<p>可以看到,最关键的就是那个 <code class="highlighter-rouge">new Function</code>。知道这步传了哪些函数进去,也就知道你的 js 脚本里都可以调用哪些内容了~</p>
<p>最后调用的 <code class="highlighter-rouge">dash_load</code> 方法也需要提一下。这个方法的最后,有几行这样的代码:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nx">self</span><span class="p">.</span><span class="nx">availablePanels</span> <span class="o">=</span> <span class="nx">_</span><span class="p">.</span><span class="nx">difference</span><span class="p">(</span><span class="nx">config</span><span class="p">.</span><span class="nx">panel_names</span><span class="p">,</span>
<span class="nx">_</span><span class="p">.</span><span class="nx">pluck</span><span class="p">(</span><span class="nx">_</span><span class="p">.</span><span class="nx">union</span><span class="p">(</span><span class="nx">self</span><span class="p">.</span><span class="nx">current</span><span class="p">.</span><span class="nx">nav</span><span class="p">,</span><span class="nx">self</span><span class="p">.</span><span class="nx">current</span><span class="p">.</span><span class="nx">pulldowns</span><span class="p">),</span><span class="s1">'type'</span><span class="p">));</span>
<span class="nx">self</span><span class="p">.</span><span class="nx">availablePanels</span> <span class="o">=</span> <span class="nx">_</span><span class="p">.</span><span class="nx">difference</span><span class="p">(</span><span class="nx">self</span><span class="p">.</span><span class="nx">availablePanels</span><span class="p">,</span><span class="nx">config</span><span class="p">.</span><span class="nx">hidden_panels</span><span class="p">);</span>
</code></pre>
</div>
<p>从最外层的 <code class="highlighter-rouge">config.js</code> 里读取了 <code class="highlighter-rouge">panel_names</code> 数组,然后取出了 nav 和 pulldown 用过的 panel,剩下就是我们能在 row 里添加的 panel 类型了。</p>
<h3 id="querysrv">querySrv</h3>
<p>querySrv.js 里定义了跟 query 框相关的函数和属性。主要有几个值得注意的。</p>
<ul>
<li>一个是 <code class="highlighter-rouge">color</code> 列表;</li>
<li>一个是 <code class="highlighter-rouge">queryTypes</code>,尤其是里么的 <code class="highlighter-rouge">topN</code>,可以看到 topN 方式其实就是先请求了一次 termsFacet,然后把结果 map 成一组普通的 query。</li>
<li>一个是 <code class="highlighter-rouge">ids</code> 和 <code class="highlighter-rouge">idsByMode</code>。之后图表的绑定具体 query 的时候,就是通过这个函数来选择的。</li>
</ul>
<h3 id="filtersrv">filterSrv</h3>
<p>filterSrv.js 跟 querySrv 相似。特殊的是两个函数。</p>
<ul>
<li>一个是 <code class="highlighter-rouge">toEjsObjs</code>。根据不同的 filter 类型调用不同的 ejs 方法。</li>
<li>一个是 <code class="highlighter-rouge">timeRange</code>。因为在 histogram panel 上拖拽,会生成好多个 range 过滤器,都是时间。这个方法会选择最后一个类型为 time 的 filter,作为实际要用的 filter。这样保证请求 ES 的是最后一次拖拽选定的时间段。</li>
</ul>
<h3 id="fields">fields</h3>
<p>fields.js 里最重要的作用就是通过 mapping 接口获取索引的字段列表,存在 <code class="highlighter-rouge">fields.list</code> 里。这个数组后来在每个 panel 的编辑页里,都以 <code class="highlighter-rouge">bs-typeahead="fields.list"</code> 的形式作为文本输入时的自动补全提示。在 table panel 里,则是左侧栏的显示来源。</p>
<h3 id="esversion">esVersion</h3>
<p>esVersion.js 里提供了对 ES 版本号的对比函数。之所以专门提供这么个 service,一来是因为不同版本的 ES 接口有变化,比如我自己开发的 percentile panel 里,就用 esVersion 判断了两次版本。因为 percentile 接口是 1.0 版之后才有,而从 1.3 版以后返回数据的结构又发生了一次变动。二来 ES 的版本号格式比较复杂,又有点又有字母。</p>
<h2 id="panel-">panel 相关指令</h2>
<h3 id="panel">添加 panel</h3>
<p>前面在讲 <code class="highlighter-rouge">app/services/dashboard.js</code> 的时候,已经说到能添加的 panel 列表是怎么获取的。那么panel 是怎么加上的呢?</p>
<p>同样是之前讲过的 <code class="highlighter-rouge">app/partials/dashaboard.html</code> 里,加载了 <code class="highlighter-rouge">partials/roweditor.html</code> 页面。这里有一段:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nt"><form</span> <span class="na">class=</span><span class="s">"form-inline"</span><span class="nt">></span>
<span class="nt"><select</span> <span class="na">class=</span><span class="s">"input-medium"</span> <span class="na">ng-model=</span><span class="s">"panel.type"</span> <span class="na">ng-options=</span><span class="s">"panelType for panelType in dashboard.availablePanels|stringSort"</span><span class="nt">></select></span>
<span class="nt"><small</span> <span class="na">ng-show=</span><span class="s">"rowSpan(row) > 11"</span><span class="nt">></span>
Note: This row is full, new panels will wrap to a new line. You should add another row.
<span class="nt"></small></span>
<span class="nt"></form></span>
<span class="nt"><div</span> <span class="na">ng-show=</span><span class="s">"!(_.isUndefined(panel.type))"</span><span class="nt">></span>
<span class="nt"><div</span> <span class="na">add-panel=</span><span class="s">""</span><span class="nt">></div></span>
<span class="nt"></div></span>
</code></pre>
</div>
<p>这个 <code class="highlighter-rouge">add-panel</code> 指令,是有 <code class="highlighter-rouge">app/directives/addPanel.js</code> 提供的。方法如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nx">$scope</span><span class="p">.</span><span class="nx">$watch</span><span class="p">(</span><span class="s1">'panel.type'</span><span class="p">,</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">_type</span> <span class="o">=</span> <span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">type</span><span class="p">;</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">reset_panel</span><span class="p">(</span><span class="nx">_type</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="o">!</span><span class="nx">_</span><span class="p">.</span><span class="nx">isUndefined</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">type</span><span class="p">))</span> <span class="p">{</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">loadingEditor</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">require</span><span class="p">([</span><span class="s1">'panels/'</span><span class="o">+</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">type</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="s2">"."</span><span class="p">,</span><span class="s2">"/"</span><span class="p">)</span> <span class="o">+</span><span class="s1">'/module'</span><span class="p">],</span> <span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">template</span> <span class="o">=</span> <span class="s1">'<div ng-controller="'</span><span class="o">+</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">type</span><span class="o">+</span><span class="s1">'" ng-include="\'app/partials/paneladd.html\'"></div>'</span><span class="p">;</span>
<span class="nx">elem</span><span class="p">.</span><span class="nx">html</span><span class="p">(</span><span class="nx">$compile</span><span class="p">(</span><span class="nx">angular</span><span class="p">.</span><span class="nx">element</span><span class="p">(</span><span class="nx">template</span><span class="p">))(</span><span class="nx">$scope</span><span class="p">));</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">loadingEditor</span> <span class="o">=</span> <span class="kc">false</span><span class="p">;</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="p">});</span>
</code></pre>
</div>
<p>可以看到,其实就是 require 了对应的 <code class="highlighter-rouge">panels/xxx/module.js</code>,然后动态生成一个 div,绑定到对应的 controller 上。</p>
<h3 id="panel-1">展示 panel</h3>
<p>还是在 <code class="highlighter-rouge">app/partials/dashaboard.html</code> 里,用到了另一个指令 <code class="highlighter-rouge">kibana-panel</code>:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nt"><div</span>
<span class="na">ng-repeat=</span><span class="s">"(name, panel) in row.panels|filter:isPanel"</span>
<span class="na">ng-cloak</span> <span class="na">ng-hide=</span><span class="s">"panel.hide"</span>
<span class="na">kibana-panel</span> <span class="na">type=</span><span class="s">'panel.type'</span> <span class="na">resizable</span>
<span class="na">class=</span><span class="s">"panel nospace"</span> <span class="na">ng-class=</span><span class="s">"{'dragInProgress':dashboard.panelDragging}"</span>
<span class="na">style=</span><span class="s">"position:relative"</span> <span class="na">ng-style=</span><span class="s">"{'width':!panel.span?'100%':((panel.span/1.2)*10)+'%'}"</span>
<span class="na">data-drop=</span><span class="s">"true"</span> <span class="na">ng-model=</span><span class="s">"row.panels"</span> <span class="na">data-jqyoui-options</span>
<span class="na">jqyoui-droppable=</span><span class="s">"{index:$index,mutate:false,onDrop:'panelMoveDrop',onOver:'panelMoveOver(true)',onOut:'panelMoveOut'}"</span><span class="nt">></span>
<span class="nt"></div></span>
</code></pre>
</div>
<p>当然,这里面还有 <code class="highlighter-rouge">resizable</code> 指令也是自己实现的,不过一般我们用不着关心这个的代码实现。</p>
<p>下面看 <code class="highlighter-rouge">app/directives/kibanaPanel.js</code> 里的实现。</p>
<p>这个里面大多数逻辑跟 addPanel.js 是一样的,都是为了实现一个指令嘛。对于我们来说,关注点在前面那一大段 HTML 字符串,也就是变量 <code class="highlighter-rouge">panelHeader</code>。这个就是我们看到的实际效果中,kibana 3 每个 panel 顶部那个小图标工具栏。仔细阅读一下,可以发现除了每个 panel 都一致的那些 span 以外,还有一段是:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="s1">'<span ng-repeat="task in panelMeta.modals" class="row-button extra" ng-show="task.show">'</span> <span class="o">+</span>
<span class="s1">'<span bs-modal="task.partial" class="pointer"><i '</span> <span class="o">+</span>
<span class="s1">'bs-tooltip="task.description" ng-class="task.icon" class="pointer"></i></span>'</span><span class="o">+</span>
<span class="s1">'</span>'</span>
</code></pre>
</div>
<p>也就是说,每个 panel 可以在自己的 panelMeta.modals 数组里,定义不同的小图标,弹出不同的对话浮层。我个人给 table panel 二次开发加入的 exportAsCsv 功能,图标就是在这里加入的。</p>
<h2 id="panel--1">panel 内部实现</h2>
<p>终于说到最后了。大家进入到 <code class="highlighter-rouge">app/panels/</code> 下,每个目录都是一种 panel。原因前一节已经分析过了,因为 addPanel.js 里就是直接这样拼接的。入口都是固定的:module.js。</p>
<p>下面以 stats panel 为例。(因为我最开始就是抄的 stats 做的 percentile,只有表格没有图形,最简单)</p>
<p>每个目录下都会有至少一下三个文件:</p>
<h3 id="modulejs">module.js</h3>
<p>module.js 就是一个 controller。跟前面讲过的 controller 写法其实是一致的。在 <code class="highlighter-rouge">$scope</code> 对象上,有几个属性是 panel 实现时一般都会有的:</p>
<ul>
<li><code class="highlighter-rouge">$scope.panelMeta</code>: 这个前面说到过,其中的 modals 用来定义 panelHeader。</li>
<li><code class="highlighter-rouge">$scope.panel</code>: 用来定义 panel 的属性。一般实现上,会有一个 default 值预定义好。你会发现这个 <code class="highlighter-rouge">$scope.panel</code> 其实就是仪表板纲要里面说的每个 panel 的可设置值!</li>
</ul>
<p>然后一般 <code class="highlighter-rouge">$scope.init()</code> 都是这样的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nx">$scope</span><span class="p">.</span><span class="nx">init</span> <span class="o">=</span> <span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">ready</span> <span class="o">=</span> <span class="kc">false</span><span class="p">;</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">$on</span><span class="p">(</span><span class="s1">'refresh'</span><span class="p">,</span> <span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">get_data</span><span class="p">();</span>
<span class="p">});</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">get_data</span><span class="p">();</span>
<span class="p">};</span>
</code></pre>
</div>
<p>也就是每次有刷新操作,就执行 <code class="highlighter-rouge">get_data()</code> 方法。这个方法就是获取 ES 数据,然后渲染效果的入口。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nx">$scope</span><span class="p">.</span><span class="nx">get_data</span> <span class="o">=</span> <span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="nx">dashboard</span><span class="p">.</span><span class="nx">indices</span><span class="p">.</span><span class="nx">length</span> <span class="o">===</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">panelMeta</span><span class="p">.</span><span class="nx">loading</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">request</span><span class="p">,</span>
<span class="nx">results</span><span class="p">,</span>
<span class="nx">boolQuery</span><span class="p">,</span>
<span class="nx">queries</span><span class="p">;</span>
<span class="nx">request</span> <span class="o">=</span> <span class="nx">$scope</span><span class="p">.</span><span class="nx">ejs</span><span class="p">.</span><span class="nx">Request</span><span class="p">();</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">queries</span><span class="p">.</span><span class="nx">ids</span> <span class="o">=</span> <span class="nx">querySrv</span><span class="p">.</span><span class="nx">idsByMode</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">queries</span><span class="p">);</span>
<span class="nx">queries</span> <span class="o">=</span> <span class="nx">querySrv</span><span class="p">.</span><span class="nx">getQueryObjs</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">queries</span><span class="p">.</span><span class="nx">ids</span><span class="p">);</span>
<span class="nx">boolQuery</span> <span class="o">=</span> <span class="nx">$scope</span><span class="p">.</span><span class="nx">ejs</span><span class="p">.</span><span class="nx">BoolQuery</span><span class="p">();</span>
<span class="nx">_</span><span class="p">.</span><span class="nx">each</span><span class="p">(</span><span class="nx">queries</span><span class="p">,</span><span class="kd">function</span><span class="p">(</span><span class="nx">q</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">boolQuery</span> <span class="o">=</span> <span class="nx">boolQuery</span><span class="p">.</span><span class="nx">should</span><span class="p">(</span><span class="nx">querySrv</span><span class="p">.</span><span class="nx">toEjsObj</span><span class="p">(</span><span class="nx">q</span><span class="p">));</span>
<span class="p">});</span>
<span class="nx">request</span> <span class="o">=</span> <span class="nx">request</span>
<span class="p">.</span><span class="nx">facet</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">ejs</span><span class="p">.</span><span class="nx">StatisticalFacet</span><span class="p">(</span><span class="s1">'stats'</span><span class="p">)</span>
<span class="p">.</span><span class="nx">field</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">field</span><span class="p">)</span>
<span class="p">.</span><span class="nx">facetFilter</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">ejs</span><span class="p">.</span><span class="nx">QueryFilter</span><span class="p">(</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">ejs</span><span class="p">.</span><span class="nx">FilteredQuery</span><span class="p">(</span>
<span class="nx">boolQuery</span><span class="p">,</span>
<span class="nx">filterSrv</span><span class="p">.</span><span class="nx">getBoolFilter</span><span class="p">(</span><span class="nx">filterSrv</span><span class="p">.</span><span class="nx">ids</span><span class="p">())</span>
<span class="p">)))).</span><span class="nx">size</span><span class="p">(</span><span class="mi">0</span><span class="p">);</span>
<span class="nx">_</span><span class="p">.</span><span class="nx">each</span><span class="p">(</span><span class="nx">queries</span><span class="p">,</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">q</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">alias</span> <span class="o">=</span> <span class="nx">q</span><span class="p">.</span><span class="nx">alias</span> <span class="o">||</span> <span class="nx">q</span><span class="p">.</span><span class="nx">query</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">query</span> <span class="o">=</span> <span class="nx">$scope</span><span class="p">.</span><span class="nx">ejs</span><span class="p">.</span><span class="nx">BoolQuery</span><span class="p">();</span>
<span class="nx">query</span><span class="p">.</span><span class="nx">should</span><span class="p">(</span><span class="nx">querySrv</span><span class="p">.</span><span class="nx">toEjsObj</span><span class="p">(</span><span class="nx">q</span><span class="p">));</span>
<span class="nx">request</span><span class="p">.</span><span class="nx">facet</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">ejs</span><span class="p">.</span><span class="nx">StatisticalFacet</span><span class="p">(</span><span class="s1">'stats_'</span><span class="o">+</span><span class="nx">alias</span><span class="p">)</span>
<span class="p">.</span><span class="nx">field</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">field</span><span class="p">)</span>
<span class="p">.</span><span class="nx">facetFilter</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">ejs</span><span class="p">.</span><span class="nx">QueryFilter</span><span class="p">(</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">ejs</span><span class="p">.</span><span class="nx">FilteredQuery</span><span class="p">(</span>
<span class="nx">query</span><span class="p">,</span>
<span class="nx">filterSrv</span><span class="p">.</span><span class="nx">getBoolFilter</span><span class="p">(</span><span class="nx">filterSrv</span><span class="p">.</span><span class="nx">ids</span><span class="p">())</span>
<span class="p">)</span>
<span class="p">))</span>
<span class="p">);</span>
<span class="p">});</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">inspector</span> <span class="o">=</span> <span class="nx">request</span><span class="p">.</span><span class="nx">toJSON</span><span class="p">();</span>
<span class="nx">results</span> <span class="o">=</span> <span class="nx">$scope</span><span class="p">.</span><span class="nx">ejs</span><span class="p">.</span><span class="nx">doSearch</span><span class="p">(</span><span class="nx">dashboard</span><span class="p">.</span><span class="nx">indices</span><span class="p">,</span> <span class="nx">request</span><span class="p">);</span>
<span class="nx">results</span><span class="p">.</span><span class="nx">then</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">results</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">panelMeta</span><span class="p">.</span><span class="nx">loading</span> <span class="o">=</span> <span class="kc">false</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">value</span> <span class="o">=</span> <span class="nx">results</span><span class="p">.</span><span class="nx">facets</span><span class="p">.</span><span class="nx">stats</span><span class="p">[</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">mode</span><span class="p">];</span>
<span class="kd">var</span> <span class="nx">rows</span> <span class="o">=</span> <span class="nx">queries</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="kd">function</span> <span class="p">(</span><span class="nx">q</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">alias</span> <span class="o">=</span> <span class="nx">q</span><span class="p">.</span><span class="nx">alias</span> <span class="o">||</span> <span class="nx">q</span><span class="p">.</span><span class="nx">query</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">obj</span> <span class="o">=</span> <span class="nx">_</span><span class="p">.</span><span class="nx">clone</span><span class="p">(</span><span class="nx">q</span><span class="p">);</span>
<span class="nx">obj</span><span class="p">.</span><span class="nx">label</span> <span class="o">=</span> <span class="nx">alias</span><span class="p">;</span>
<span class="nx">obj</span><span class="p">.</span><span class="nx">Label</span> <span class="o">=</span> <span class="nx">alias</span><span class="p">.</span><span class="nx">toLowerCase</span><span class="p">();</span> <span class="c1">//sort field</span>
<span class="nx">obj</span><span class="p">.</span><span class="nx">value</span> <span class="o">=</span> <span class="nx">results</span><span class="p">.</span><span class="nx">facets</span><span class="p">[</span><span class="s1">'stats_'</span><span class="o">+</span><span class="nx">alias</span><span class="p">];</span>
<span class="nx">obj</span><span class="p">.</span><span class="nx">Value</span> <span class="o">=</span> <span class="nx">results</span><span class="p">.</span><span class="nx">facets</span><span class="p">[</span><span class="s1">'stats_'</span><span class="o">+</span><span class="nx">alias</span><span class="p">];</span> <span class="c1">//sort field</span>
<span class="k">return</span> <span class="nx">obj</span><span class="p">;</span>
<span class="p">});</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">data</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">value</span><span class="p">:</span> <span class="nx">value</span><span class="p">,</span>
<span class="na">rows</span><span class="p">:</span> <span class="nx">rows</span>
<span class="p">};</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">$emit</span><span class="p">(</span><span class="s1">'render'</span><span class="p">);</span>
<span class="p">});</span>
<span class="p">};</span>
</code></pre>
</div>
<p>stats panel 的这段函数几乎就跟基础示例一样了。</p>
<ol>
<li>生成 Request 对象。</li>
<li>获取关联的 query 对象。</li>
<li>获取当前页的 filter 对象。</li>
<li>调用选定的 facets 方法,传入参数。</li>
<li>如果有多个 query,逐一构建 facets。</li>
<li>request 完成。生成一个 JSON 内容供 inspector 查看。</li>
<li>发送请求,等待异步回调。</li>
<li>回调处理数据成绑定在模板上的 <code class="highlighter-rouge">$scope.data</code>。</li>
<li>渲染页面。</li>
</ol>
<p>注:stats/module.js 后面还有一个 filter,terms/module.js 后面还有一个 directive,这些都是为了实际页面效果加的功能,跟 kibana 本身的 filter,directive 本质上是一样的。就不单独讲述了。</p>
<h3 id="modulehtml">module.html</h3>
<p>module.html 就是 panel 的具体页面内容。没有太多可说的。大概框架是:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nt"><div</span> <span class="na">ng-controller=</span><span class="s">'stats'</span> <span class="na">ng-init=</span><span class="s">"init()"</span><span class="nt">></span>
<span class="nt"><table</span> <span class="na">ng-style=</span><span class="s">"panel.style"</span> <span class="na">class=</span><span class="s">"table table-striped table-condensed"</span> <span class="na">ng-show=</span><span class="s">"panel.chart == 'table'"</span><span class="nt">></span>
<span class="nt"><thead></span>
<span class="nt"><th></span>Term<span class="nt"></th></span> <span class="nt"><th></th></span> <span class="nt"><th></span>Action<span class="nt"></th></span>
<span class="nt"></thead></span>
<span class="nt"><tr</span> <span class="na">ng-repeat=</span><span class="s">"term in data"</span> <span class="na">ng-show=</span><span class="s">"showMeta(term)"</span><span class="nt">></span>
<span class="nt"><td</span> <span class="na">class=</span><span class="s">"terms-legend-term"</span><span class="nt">></td></span>
<span class="nt"><td></td></span>
<span class="nt"></tr></span>
<span class="nt"></table></span>
<span class="nt"></div></span>
</code></pre>
</div>
<p>主要就是绑定要 controller 和 init 函数。对于示例的 stats,里面的 <code class="highlighter-rouge">data</code> 就是 module.js 最后生成的 <code class="highlighter-rouge">$scope.data</code>。</p>
<h3 id="editorhtml">editor.html</h3>
<p>editor.html 是 panel 参数的编辑页面主要内容,参数编辑还有一些共同的标签页,是在 kibana 的 <code class="highlighter-rouge">app/partials/</code> 里,就不讲了。</p>
<p>editor.html 里,主要就是提供对 <code class="highlighter-rouge">$scope.panel</code> 里那些参数的修改保存操作。当然实际上并不是所有参数都暴露出来了。这也是 kibana 3 用户指南里,官方说采用仪表板纲要,比通过页面修改更灵活细腻的原因。</p>
<p>editor.html 里需要注意的是,为了每次变更都能实时生效,所有的输入框都注册到了刷新事件。所以一般是这样子:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nt"><select</span> <span class="na">ng-change=</span><span class="s">"set_refresh(true)"</span> <span class="na">class=</span><span class="s">"input-small"</span> <span class="na">ng-model=</span><span class="s">"panel.format"</span> <span class="na">ng-options=</span><span class="s">"f for f in ['number','float','money','bytes']"</span><span class="nt">></select></span>
</code></pre>
</div>
<p>这个 <code class="highlighter-rouge">set_refresh</code> 函数是在 <code class="highlighter-rouge">module.js</code> 里定义的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nx">$scope</span><span class="p">.</span><span class="nx">set_refresh</span> <span class="o">=</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">state</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">refresh</span> <span class="o">=</span> <span class="nx">state</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<h2 id="section-2">总结</h2>
<p>kibana 3 源码的主体分析,就是这样了。怎么样,看完以后,大家有没有信心也做些二次开发,甚至跟 grafana 一样,替换掉 esResource,换上一个你自己的后端数据源呢?</p>
用 Kibana4 实现 PHP 慢日志函数堆栈分析
2015-03-06T00:00:00+08:00
logstash
kibana
php
logstash
python
http://chenlinux.com/2015/03/06/kibana4-for-slowlog
<p>标题说是 PHP 的慢日志,其实所有函数堆栈的调试日志都可以做,比如 Java 的调试日志等等。要用 Kibana ,首先得把日志数据解析并输入到 Elasticsearch 里。所以,本文分为几个部分:多行合并,堆栈解析,Nested Aggs 处理,Kibana4 的可视化效果。</p>
<h2 id="section">多行合并</h2>
<p>堆栈日志显然都是多行的。所以首先需要把多行数据整合成单个事件。之前已经多次写过如何<a href="https://github.com/chenryn/logstash-best-practice-cn/blob/master/codec/multiline.md">用 Logstash 实现这个需求</a>了。不过,Logstash 这里有个限制,就是必须是在 shipper 段配置才能有用。如果在 index 端,不同 shipper 来的数据顺序已经打乱了,这个合并就没有意义了。</p>
<p>所以,如果日志收集的时候没有用 Logstash 的,这时候就得自己处理了。下面是我写的一个示例:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#!/usr/bin/env pypy</span>
<span class="c">#coding:utf-8</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">socket</span>
<span class="kn">import</span> <span class="nn">urllib2</span>
<span class="kn">import</span> <span class="nn">optparse</span>
<span class="k">try</span><span class="p">:</span> <span class="kn">import</span> <span class="nn">simplejson</span> <span class="kn">as</span> <span class="nn">json</span>
<span class="k">except</span> <span class="nb">ImportError</span><span class="p">:</span> <span class="kn">import</span> <span class="nn">json</span>
<span class="kn">from</span> <span class="nn">common</span> <span class="kn">import</span> <span class="n">grokFpmSlow</span>
<span class="n">defaultLogTag</span><span class="o">=</span><span class="s">'fpmSlow'</span>
<span class="n">hostname</span> <span class="o">=</span> <span class="n">socket</span><span class="o">.</span><span class="n">gethostname</span><span class="p">()</span>
<span class="n">timeout</span> <span class="o">=</span> <span class="mi">120</span>
<span class="k">def</span> <span class="nf">getOptions</span><span class="p">():</span>
<span class="n">usage</span> <span class="o">=</span> <span class="s">"usage: </span><span class="si">%</span><span class="s">prog [options]"</span>
<span class="n">OptionParser</span> <span class="o">=</span> <span class="n">optparse</span><span class="o">.</span><span class="n">OptionParser</span>
<span class="n">parser</span> <span class="o">=</span> <span class="n">OptionParser</span><span class="p">(</span><span class="n">usage</span><span class="p">)</span>
<span class="n">parser</span><span class="o">.</span><span class="n">add_option</span><span class="p">(</span><span class="s">"-t"</span><span class="p">,</span><span class="s">"--logTag"</span><span class="p">,</span><span class="n">action</span><span class="o">=</span><span class="s">"store"</span><span class="p">,</span><span class="nb">type</span><span class="o">=</span><span class="s">"string"</span><span class="p">,</span><span class="n">dest</span><span class="o">=</span><span class="s">"logTag"</span><span class="p">,</span><span class="n">default</span><span class="o">=</span><span class="n">defaultLogTag</span><span class="p">,</span><span class="n">help</span><span class="o">=</span><span class="s">"default log tag."</span><span class="p">)</span>
<span class="n">options</span><span class="p">,</span><span class="n">args</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_args</span><span class="p">()</span>
<span class="k">return</span> <span class="n">options</span><span class="p">,</span><span class="n">args</span>
<span class="k">def</span> <span class="nf">send_es</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">logtag</span><span class="p">):</span>
<span class="n">url</span> <span class="o">=</span> <span class="s">'http://esdomain:9200/logstash-mweibo-'</span> <span class="o">+</span> <span class="n">data</span><span class="p">[</span><span class="s">'@timestamp'</span><span class="p">]</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">'T'</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s">'-'</span><span class="p">,</span><span class="s">'.'</span><span class="p">)</span> <span class="o">+</span> <span class="s">'/'</span> <span class="o">+</span> <span class="n">logtag</span>
<span class="n">req</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">data</span><span class="p">),</span> <span class="p">{</span><span class="s">'Content-Type'</span><span class="p">:</span><span class="s">'application/json'</span><span class="p">})</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">res</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span>
<span class="k">print</span> <span class="s">"Return content:"</span><span class="p">,</span><span class="n">res</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="k">except</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">URLError</span><span class="p">,</span> <span class="n">e</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">e</span><span class="p">,</span><span class="s">"reason"</span><span class="p">):</span>
<span class="k">if</span> <span class="n">e</span><span class="o">.</span><span class="n">reason</span> <span class="o">==</span> <span class="s">'Bad Request'</span> <span class="ow">and</span> <span class="n">data</span><span class="o">.</span><span class="n">has_key</span><span class="p">(</span><span class="s">'jsoncontent'</span><span class="p">):</span>
<span class="n">data</span><span class="p">[</span><span class="s">'message'</span><span class="p">]</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s">'jsoncontent'</span><span class="p">))</span>
<span class="n">send_es</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">logtag</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">print</span> <span class="s">"The reason:"</span><span class="p">,</span><span class="n">e</span><span class="o">.</span><span class="n">reason</span>
<span class="k">elif</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">e</span><span class="p">,</span><span class="s">"code"</span><span class="p">):</span>
<span class="k">print</span> <span class="s">"Error code:"</span><span class="p">,</span><span class="n">e</span><span class="o">.</span><span class="n">code</span>
<span class="k">print</span> <span class="s">"Return content:"</span><span class="p">,</span><span class="n">e</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">flush</span><span class="p">(</span><span class="n">log_buffer</span><span class="p">,</span> <span class="n">grokObj</span><span class="p">):</span>
<span class="n">data</span> <span class="o">=</span> <span class="s">""</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">log_buffer</span><span class="p">)</span>
<span class="n">match</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">grokObj</span><span class="o">.</span><span class="n">msg_regexp</span><span class="p">,</span> <span class="n">data</span><span class="p">)</span>
<span class="k">if</span> <span class="n">match</span><span class="p">:</span>
<span class="n">ret</span> <span class="o">=</span> <span class="n">grokObj</span><span class="o">.</span><span class="n">grokData</span><span class="p">(</span><span class="n">match</span><span class="p">)</span>
<span class="n">ret</span><span class="p">[</span><span class="s">"host"</span><span class="p">]</span> <span class="o">=</span> <span class="n">hostname</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">ret</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">"host"</span><span class="p">:</span><span class="n">hostname</span><span class="p">,</span>
<span class="s">"message"</span><span class="p">:</span><span class="n">data</span><span class="p">,</span>
<span class="s">"@timestamp"</span><span class="p">:</span><span class="n">time</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s">'</span><span class="si">%</span><span class="s">FT</span><span class="si">%</span><span class="s">T'</span><span class="p">)</span><span class="o">+</span><span class="s">'+0800'</span>
<span class="p">}</span>
<span class="n">send_es</span><span class="p">(</span><span class="n">ret</span><span class="p">,</span> <span class="n">grokObj</span><span class="o">.</span><span class="n">logtag</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">get_log</span><span class="p">(</span><span class="n">grokObj</span><span class="p">):</span>
<span class="n">start_time</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
<span class="n">log_buffer</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">line</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span>
<span class="k">except</span><span class="p">:</span>
<span class="k">break</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">line</span><span class="p">:</span>
<span class="n">flush</span><span class="p">(</span><span class="n">log_buffer</span><span class="p">,</span> <span class="n">grokObj</span><span class="p">)</span>
<span class="k">break</span>
<span class="k">if</span> <span class="n">line</span><span class="p">:</span>
<span class="n">match</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">grokObj</span><span class="o">.</span><span class="n">start_regexp</span><span class="p">,</span> <span class="n">line</span><span class="p">)</span>
<span class="k">if</span> <span class="n">match</span> <span class="ow">and</span> <span class="nb">len</span><span class="p">(</span><span class="n">log_buffer</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
<span class="n">flush</span><span class="p">(</span><span class="n">log_buffer</span><span class="p">,</span> <span class="n">grokObj</span><span class="p">)</span>
<span class="n">start_time</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
<span class="n">log_buffer</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">log_buffer</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">rstrip</span><span class="p">())</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">if</span> <span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">-</span> <span class="n">startime</span> <span class="o">></span> <span class="n">timeout</span> <span class="p">):</span>
<span class="n">flush</span><span class="p">(</span><span class="n">log_buffer</span><span class="p">,</span> <span class="n">grokObj</span><span class="p">)</span>
<span class="n">start_time</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
<span class="n">log_buffer</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">options</span><span class="p">,</span><span class="n">args</span> <span class="o">=</span> <span class="n">getOptions</span><span class="p">()</span>
<span class="k">if</span> <span class="n">options</span><span class="o">.</span><span class="n">logTag</span> <span class="o">==</span> <span class="s">''</span><span class="p">:</span>
<span class="n">get_log</span><span class="p">(</span><span class="n">grokFpmSlow</span><span class="o">.</span><span class="n">fpmSlow</span><span class="p">(</span><span class="n">options</span><span class="o">.</span><span class="n">logTag</span><span class="p">))</span>
</code></pre>
</div>
<p>python 水平很烂,大家看看就好,大概流程其实跟 Logstash 差不多。</p>
<h2 id="section-1">堆栈解析</h2>
<p>上面的 python 脚本,只是做到根据正则表达式合并多行数据,以及收到处理结果后发送给 ES 集群。具体的处理,则在 common/grokFpmSlow.py 中完成:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#/usr/bin/pypy</span>
<span class="c">#coding:utf-8</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="kn">import</span> <span class="nn">datetime</span>
<span class="k">class</span> <span class="nc">fpmSlow</span><span class="p">():</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">_logtag</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">logtag</span> <span class="o">=</span> <span class="n">_logtag</span>
<span class="bp">self</span><span class="o">.</span><span class="n">start_regexp</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="nb">compile</span><span class="p">(</span><span class="s">'^</span><span class="err">\</span><span class="s">[</span><span class="err">\</span><span class="s">d{2}-</span><span class="err">\</span><span class="s">w{3}-</span><span class="err">\</span><span class="s">d{4}'</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">msg_regexp</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="nb">compile</span><span class="p">(</span><span class="s">'(?m)</span><span class="err">\</span><span class="s">[(?P<timestamp></span><span class="err">\</span><span class="s">d{2}-</span><span class="err">\</span><span class="s">w{3}-</span><span class="err">\</span><span class="s">d{4} </span><span class="err">\</span><span class="s">d{2}:</span><span class="err">\</span><span class="s">d{2}:</span><span class="err">\</span><span class="s">d{2})</span><span class="err">\</span><span class="s">] </span><span class="err">\</span><span class="s">[pool (?P<pool></span><span class="err">\</span><span class="s">S+)</span><span class="err">\</span><span class="s">] pid (?P<pid></span><span class="err">\</span><span class="s">d+)script_filename = (?P<slow_script></span><span class="err">\</span><span class="s">S+)(?P<message></span><span class="err">\</span><span class="s">[</span><span class="err">\</span><span class="s">w{18}</span><span class="err">\</span><span class="s">] (?P<slow_func>[^</span><span class="err">\</span><span class="s">[]*?:</span><span class="err">\</span><span class="s">d+).*</span><span class="err">\</span><span class="s">[</span><span class="err">\</span><span class="s">w{18}</span><span class="err">\</span><span class="s">](?P<begin_func>[^</span><span class="err">\</span><span class="s">[]*?:</span><span class="err">\</span><span class="s">d+))$'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">grokData</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span><span class="n">match</span><span class="p">):</span>
<span class="n">ret</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="n">groupdict</span><span class="p">()</span>
<span class="n">ret</span><span class="p">[</span><span class="s">'slow'</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span><span class="n">k</span><span class="p">:</span> <span class="n">v</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">re</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">r'</span><span class="err">\</span><span class="s">[</span><span class="err">\</span><span class="s">w{18}</span><span class="err">\</span><span class="s">] '</span><span class="p">,</span> <span class="n">ret</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s">'message'</span><span class="p">)))</span> <span class="k">if</span> <span class="n">k</span> <span class="o">></span> <span class="mi">0</span> <span class="p">}</span>
<span class="n">ret</span><span class="p">[</span><span class="s">"@timestamp"</span><span class="p">]</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">ret</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s">"timestamp"</span><span class="p">),</span> <span class="s">"</span><span class="si">%</span><span class="s">d-</span><span class="si">%</span><span class="s">b-</span><span class="si">%</span><span class="s">Y </span><span class="si">%</span><span class="s">H:</span><span class="si">%</span><span class="s">M:</span><span class="si">%</span><span class="s">S"</span><span class="p">)</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s">"</span><span class="si">%</span><span class="s">FT</span><span class="si">%</span><span class="s">T+0800"</span><span class="p">)</span>
<span class="k">return</span> <span class="n">ret</span>
</code></pre>
</div>
<p>类属性中的 <code class="highlighter-rouge">start_regexp</code> 对应 Logstash/Codecs/MultiLine 中的 pattern,<code class="highlighter-rouge">msg_regexp</code> 对应 Logstash/Filters/Grok 中的 match。这些都是标准的正则,根据日志的实际情况写就好了。</p>
<p><code class="highlighter-rouge">grokData()</code> 方法里,把 <code class="highlighter-rouge">message</code> 里存的整个堆栈,首先切割成数组,然后转换成对应行号为键的字典,存入 <code class="highlighter-rouge">slow</code> 字段。</p>
<p>也就是说,原本一段这样的 PHP-FPM 慢日志:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[13-May-2013 05:17:12] [pool www] pid 13557
script_filename = /opt/www/inkebook/index.php
[0x000000000292e0f0] commit() /opt/www/inkebook/includes/database/mysql/database.inc:166
[0x000000000292de88] popCommittableTransactions() /opt/www/inkebook/includes/database/database.inc:1128
[0x000000000292dcf0] popTransaction() /opt/www/inkebook/includes/database/database.inc:1905
[0x00007fffe78cc460] __destruct() unknown:0
[0x000000000292c690] execute() /opt/www/inkebook/modules/statistics/statistics.module:73
[0x00007fffe78cc900] statistics_exit() unknown:0
[0x000000000292c208] call_user_func_array() /opt/www/inkebook/includes/module.inc:857
[0x000000000292bf10] module_invoke_all() /opt/www/inkebook/includes/common.inc:2688
[0x000000000292ade0] drupal_page_footer() /opt/www/inkebook/includes/common.inc:2676
[0x000000000292aa28] drupal_deliver_html_page() /opt/www/inkebook/includes/common.inc:2560
[0x000000000292a378] drupal_deliver_page() /opt/www/inkebook/includes/menu.inc:532
[0x000000000292a198] menu_execute_active_handler() /opt/www/inkebook/index.php:21
</code></pre>
</div>
<p>会转换成下面这样的字典:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span>
<span class="s">"pool"</span><span class="p">:</span> <span class="s">"www"</span><span class="p">,</span>
<span class="s">"pid"</span><span class="p">:</span> <span class="s">"13557"</span><span class="p">,</span>
<span class="s">"slow_script"</span><span class="p">:</span> <span class="s">"/opt/www/inkebook/index.php"</span><span class="p">,</span>
<span class="s">"slow_func"</span><span class="p">:</span> <span class="s">"commit() /opt/www/inkebook/includes/database/mysql/database.inc:166"</span><span class="p">,</span>
<span class="s">"begin_func"</span><span class="p">:</span> <span class="s">"menu_execute_active_handler() /opt/www/inkebook/index.php:21"</span><span class="p">,</span>
<span class="s">"@timestamp"</span><span class="p">:</span> <span class="s">"2013-05-13T05:17:12+0800"</span><span class="p">,</span>
<span class="s">"slow"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"1"</span><span class="p">:</span> <span class="s">"commit() /opt/www/inkebook/includes/database/mysql/database.inc:166"</span><span class="p">,</span>
<span class="s">"2"</span><span class="p">:</span> <span class="s">"popCommittableTransactions() /opt/www/inkebook/includes/database/database.inc:1128"</span><span class="p">,</span>
<span class="s">"3"</span><span class="p">:</span> <span class="s">"popTransaction() /opt/www/inkebook/includes/database/database.inc:1905"</span><span class="p">,</span>
<span class="s">"4"</span><span class="p">:</span> <span class="s">"__destruct() unknown:0"</span><span class="p">,</span>
<span class="s">"5"</span><span class="p">:</span> <span class="s">"execute() /opt/www/inkebook/modules/statistics/statistics.module:73"</span><span class="p">,</span>
<span class="s">"6"</span><span class="p">:</span> <span class="s">"statistics_exit() unknown:0"</span><span class="p">,</span>
<span class="s">"7"</span><span class="p">:</span> <span class="s">"call_user_func_array() /opt/www/inkebook/includes/module.inc:857"</span><span class="p">,</span>
<span class="s">"8"</span><span class="p">:</span> <span class="s">"module_invoke_all() /opt/www/inkebook/includes/common.inc:2688"</span><span class="p">,</span>
<span class="s">"9"</span><span class="p">:</span> <span class="s">"drupal_page_footer() /opt/www/inkebook/includes/common.inc:2676"</span><span class="p">,</span>
<span class="s">"10"</span><span class="p">:</span> <span class="s">"drupal_deliver_html_page() /opt/www/inkebook/includes/common.inc:2560"</span><span class="p">,</span>
<span class="s">"11"</span><span class="p">:</span> <span class="s">"drupal_deliver_page() /opt/www/inkebook/includes/menu.inc:532"</span><span class="p">,</span>
<span class="s">"12"</span><span class="p">:</span> <span class="s">"menu_execute_active_handler() /opt/www/inkebook/index.php:21"</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>现在,数据就算处理完毕,可以写入 ES 了。</p>
<h2 id="nested-aggs">Nested Aggs</h2>
<p>Elasticsearch 从 1.0 版本开始,改用 Agg 替换了 Facet 接口。其中最重要的特性,就是 Agg 可以叠加。还是以本文为例。因为我们只需要对函数做 Terms Agg 计数,所以 Nested Aggs 都是“桶(bucket)”类型的聚合。Elasticsearch 先按照第一级聚合的要求划分数据到桶内,也就是按照 <code class="highlighter-rouge">slow.1</code> 的 TopN 划成 10 个桶;然后在这 10 个桶内,按照第二级聚合的要求再划分数据到第二级桶,也就是在前面 10 个桶里按照 <code class="highlighter-rouge">slow.2</code> 的 TopN 各自又划分成 10 个桶,以此类推。</p>
<p><em>Elasticsearch 除了 bucket 类型的聚合还有 metric 类型的聚合。其实,如果一个 Terms Agg 叠加一个 metric 类型聚合的效果,就跟 Kibana 3 里的 TopN query 效果类似。但是 Nested Aggs 即可以叠加 metric 也可以叠加 bucket 类型的聚合,而且还可以叠加不止一次,功能更加强大。另外,Nested Aggs 是一次请求,Elasticsearch 全部计算完成统一返回。而 Kibana 3 里的效果其实是单独请求一次 TopN,然后循环发起 N 次带有 terms filter 的 facet 请求。</em></p>
<p>关于 Nested Agg 在叠加时候次序的影响,可以参见前不久我翻译的官网博客<a href="/2015/02/25/kibana-aggregation-execution-order-and-you/">《kibana 的聚合执行次序》</a>一文,颠倒 <code class="highlighter-rouge">terms</code> 和 <code class="highlighter-rouge">date_histogram</code> 的叠加次序,需求和结果是不一样的。</p>
<h2 id="section-2">效果图</h2>
<p>好了,铺垫完成了。终于说到 Kibana 4 里的操作了。</p>
<p>这次用的是 Kibana 4 正式版。也就是改用了 nodejs 的版本。所以,运行是很简单了。如果是下的压缩包,解压开,修改好 <code class="highlighter-rouge">config.yml</code> 运行 <code class="highlighter-rouge">bin/kibana</code> 即可。如果是用的 git 仓库源码,运行 <code class="highlighter-rouge">npm install && npm start</code> 即可。</p>
<p><em>正式版要求 ES 版本是 1.4.4,如果你是 1.4.0 ~ 1.4.3 的,这几个版本之间没有功能区别,只是那个脚本沙箱的漏洞。可以直接修改 <code class="highlighter-rouge">src/public/index.js</code>(源码则是 <code class="highlighter-rouge">src/kibana/index.js</code>) 里版本判断那行代码。</em></p>
<p><em>Kibana 4 里还会检查 ES 集群的分片状态,如果有 INIT 状态的分片,直接连 server 都不会启动,一定要等待集群完全 green 了才行。这是个很没道理的做法。我只要每个号有一个分片能用,就不影响数据读取啊!碰巧也有这个问题的,可以修改 <code class="highlighter-rouge">src/server/lib/waitForEs.js</code> 里的 <code class="highlighter-rouge">waitForShards()</code> 函数,直接强制 return 即可。实测完全没影响。</em></p>
<p>运行起来以后,访问主机的 5601 端口,就可以打开 Kibana4 的页面了。配置索引模式等步骤,这里不详说,可以参见我刚翻译完的<a href="http://kibana.logstash.es/content/v4/README.html">《Kibana 4 用户指南》</a>。</p>
<p>总之,在 Discover 页添加一个 query 或者 filter,目的是过滤出来 php-fpm 的 slow 日志数据,完成后保存,命名。</p>
<p>然后进 Visualize 页,添加一个 pie chart。选择 aggregation 类型为 terms。选择字段为 <code class="highlighter-rouge">slow.1</code>(如果采用了类 logstash 的template,这里应该用 <code class="highlighter-rouge">slow.1.raw</code> 确保函数不会被分词)。然后点击 <code class="highlighter-rouge">split slices</code>,继续添加 aggregation,这次字段为 <code class="highlighter-rouge">slow.2</code>。以此类推,假设我们一直添加到了 <code class="highlighter-rouge">slow.4</code>。</p>
<p>好了,页面右侧出现了最终的效果:</p>
<p><img src="/images/uploads/k4-split-bucket-pie.jpg" alt="split slices pie chart" /></p>
<p>点击保存,输入命名。之后,可以在 Dashboard 页加入这个图片,也可以直接在其他页面里嵌入这个图片。点击 share 图标就可以看到 URL 了。如下:</p>
<blockquote>
<p>http://sla.weibo.cn:5601/#/visualize/edit/php-slow-stack-pie?embed&_g=(time:(from:now-24h,mode:quick,to:now))&_a=(filters:!(),linked:!t,query:(query_string:(query:’*’)),vis:(aggs:!((id:’1’,params:(),schema:metric,type:count),(id:’2’,params:(field:slow.1.raw,order:desc,orderBy:’1’,size:10),schema:segment,type:terms),(id:’3’,params:(field:slow.2.raw,order:desc,orderBy:’1’,size:10),schema:segment,type:terms),(id:’4’,params:(field:slow.3.raw,order:desc,orderBy:’1’,size:10),schema:segment,type:terms),(id:’5’,params:(field:slow.4.raw,order:desc,orderBy:’1’,size:10),schema:segment,type:terms)),listeners:(),params:(addLegend:!t,addTooltip:!t,defaultYExtents:!f,isDonut:!t,shareYAxis:!t,spyPerPage:10),type:pie))</p>
</blockquote>
<p>这个 URL 设计也是 Kibana 4 的一个重大改进之一。可以看到,基本上大多数设置都在这个 urlparams 里了。这也就意味着,我们其实可以直接修改 URL 来达到快速变换效果的目的。比如,我们现在想看到 <code class="highlighter-rouge">slow.5</code> 的效果,只需要在 URL 里加上一段 <code class="highlighter-rouge">(id:'6',params:(field:slow.5.raw,order:desc,orderBy:'1',size:10),</code> 就完工了。要改看两天的分析数据,只需要修改 URL 里的 <code class="highlighter-rouge">(time:(from:now-2d,mode:quick,to:now))</code> 就可以了。想恢复编辑页面而不是内嵌图片形式,把 URL 里的 <code class="highlighter-rouge">embed&</code> 去掉就可以了。</p>
<p>事实上,掌握 URL 方式非常有用!因为 Kibana 4 中,Visualize 页的字段都是下拉菜单选择的方式,不像 Kibana 3 里是文本框任意输入。菜单选择方式,可以根据聚合的要求过滤不符合要求的类型的字段,一般来说是更方便的。但是:如果你的数据量很大,结构很复杂,可能这个下拉菜单你滚轴滚上几十秒都找不到想要的字段(因为需要提前准备好字段的细节,Kibana 4 在初次访问的时候会从 ES 下载当前索引模式下整个的字段映射数据<code class="highlighter-rouge">/_mapping/fields/*</code>,字段一多,这个数据就很大,又要保存在浏览器内存里,可以想象浏览器会多卡)!我的实际环境中,有 22000+ 个字段,映射请求的响应体大小高达 70MB,最后只好放弃在菜单里寻找需要的字段,随意选了一个,然后在 URL 里改成自己要的……</p>
<p>btw: 以上在 safari 上正常完成,在 chrome 40 上有 “Maximum call stack size exceeded” 报错,尚不知道根源。</p>
【翻译】kibana 的聚合执行次序
2015-02-25T00:00:00+08:00
logstash
kibana
elasticsearch
http://chenlinux.com/2015/02/25/kibana-aggregation-execution-order-and-you
<p>原文地址:<a href="http://www.elasticsearch.org/blog/kibana-aggregation-execution-order-and-you/">http://www.elasticsearch.org/blog/kibana-aggregation-execution-order-and-you/</a></p>
<p>可能现在你已经发现了 Kibana 4 的 Visualize 界面上那些狡猾的小箭头,然后会问:“你们在那干嘛呢?有啥用啊?”嗯,这些按钮是用来控制聚合执行次序的。这个就定义了 Elasticsearch 如何分析你的数据,以及 Kibana 如何展示结果。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/crafty_arrows.png" alt="" /></p>
<p>让我们预设一个常见的场景:按时序查找最活跃的用户。很简单对吧?没错,不过其实你的需求并不明确,目的并不清楚。什么叫“最活跃的用户”?让我们多加几个参数:一年时间,按照每周,计算前 5 名用户。现在更接近结果了,不过我们还是有两条不同的方式来解释这个需求:</p>
<ol>
<li>一年时间内的前 5 名用户,他们的每周活跃度</li>
<li>每周的前 5 名用户,持续统计一年</li>
</ol>
<h2 id="section">每周的前 5 名用户,持续统计一年</h2>
<p>这个截屏里,我们先运行时间轴柱状图(date histogram),然后再问前 5 名用户。这就会给一年的每个星期创建一个桶(bucket,译者注:ES 的 聚合 API 响应内容就是以 bucket 存在的)。在每个星期里,我们找到前 5 名用户,所以在这种情况下,每周的前 5 名用户,可能都是不一样的,最后在图例里,你就看到超过 5 个用户了。</p>
<p>然后,如果我们点开阴影区域的聚合请求(Request)标签,可以看到,date histogram 是先请求的,在 date histogram 里再加上了 terms 的聚合。结果就是我们看到有些星期某些用户异常活跃,而他们可能在其他时候毫无动静。这样我们就找到指定星期里的离群数据了。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/Screen-Shot-2015-02-19-at-4.12.38-PM-1024x751.png" alt="" /></p>
<h2 id="section-1">一年时间内的前 5 名用户,他们的每周活跃度</h2>
<p>现在,我们点击向上箭头,把 terms 聚合移到 date histogram 上面来。现在我们是先计算整年的前 5 名用户,然后给每个用户创建一个 date histogram。这下图例里就只有 5 个值了。不过,我们现在看到的用户也是持续活跃的,不再有离群数据了。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/Screen-Shot-2015-02-19-at-4.12.55-PM-1024x750.png" alt="" /></p>
<h2 id="section-2">总结</h2>
<p>所以现在你知道了:这些箭头还是有用的。聚合执行次序应用于 Kibana 里几乎所有的图,所以显著影响着你在图上看到的数据,以及你从数据得出的结论。</p>
<p>最后,如果你觉得自己有关于 Kibana 的好故事,我们很乐意倾听。发邮件到 <a href="stories@elasticsearch.com">stories@elasticsearch.com</a> 或者 <a href="http://www.twitter.com/elasticsearch">在 Twitter</a> 上联系,我们会帮你分享成功的喜悦给全世界!</p>
【翻译】Kibana 4 RC1 发布
2015-02-25T00:00:00+08:00
logstash
kibana
http://chenlinux.com/2015/02/25/kibana-4-rc1-is-now-available
<p>原文地址:<a href="http://www.elasticsearch.org/blog/kibana-4-rc1-is-now-available">http://www.elasticsearch.org/blog/kibana-4-rc1-is-now-available</a></p>
<p>Kibana 4 的第一个 RC 版带着可选色、可堆叠、柱状图、饼图等等来啦!你应该注意到标题里的字母了,没错,现在不再是 beta 了。这意味着什么?这意味着我们打磨好了毛边,擦干净了痕迹。也意味着更加稳定,更好的性能,以及一些新的特性。</p>
<p>The good stuff is below, but if you want to jump right in then upgrade to Elasticsearch 1.4.3 and grab the new build over on the <a href="http://www.elasticsearch.org/overview/kibana/installation/">Kibana 4 download page</a> right away.</p>
<p><strong>小贴士</strong></p>
<ol>
<li>建议升级到 Elasticsearch 1.4.3。 Kibana 4 RC1 依赖一些 Elasticsearch 1.4.3 的功能。</li>
<li>更新你的 <code class="highlighter-rouge">kibana.yml</code>,有些配置参数发生了变化,比如 <code class="highlighter-rouge">elasticsearch</code> 现在叫 <code class="highlighter-rouge">elasticsearch_url</code>。</li>
</ol>
<h2 id="section">多序列图</h2>
<p>Kibana 4 现在支持在每个图上画多个数值聚合。比如,在一个图上显示一个字段,或者完全不相关的多个字段的最小、最大和平均值。我们还添加上了呼声很高的百分比聚合,以及标准差视图。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/Screen-Shot-2015-02-11-at-4.47.29-PM-1024x572.png" alt="" /></p>
<h2 id="section-1">部分数据桶的标示</h2>
<p>你可能注意到过,很多分析引擎的最后一个点上,数据总是下降的。这是因为最后一个条带本身“没满”。比如一个每天的条带图,但是今天还没结束呢。Kibana 现在会给你展示这一天还有多少剩余时间,通过一个微妙的阴影设计,表示还有后续的时间序列数据。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/Screen-Shot-2015-02-10-at-8.20.44-AM-1024x573.png" alt="" /></p>
<h2 id="section-2">仪表板上的文档表格</h2>
<p>作为可视化的补充,Kibana 现在也可以在仪表板上展示已存的搜索了。和添加可视化内容一样操作,不过注意这个 “Searches” 标签。Kibana 会加载你保存的搜索,包括它的各列内容,然后排序列入仪表板上的表格。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/Screen-Shot-2015-02-11-at-4.53.55-PM-1024x633.png" alt="" /></p>
<h2 id="markdown-">markdown 挂件和表格过滤器</h2>
<p>是不是厌烦饿了回答这个问题:“这行是啥意思?”。markdown 挂件让你可以给复杂的仪表板添加帮助信息面板。而且,数据表格现在跟其他面板一样也支持点击生成过滤器的功能了。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/Screen-Shot-2015-02-11-at-9.13.34-PM-1024x677.png" alt="" /></p>
<h2 id="section-3">脚本化字段上的过滤器</h2>
<p>Beta 3 不允许在脚本上做过滤。RC 现在通过透明传输的方式支持了 Elasticsearch 的 script filter 功能。在脚本化字段上点击生成过滤器,就跟普通字段一样。</p>
<p>Kibana 4 RC1 同时还从 Groovy 迁移到了 <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_lucene_expressions_scripts">Lucene Expressions</a>,这个变化出自 Elasticsearch 1.4.3 版的变更。因为 Lucene expressions 目前只支持数值类型的数据和函数,我们正在努力,早日支持字符串、时间类型。</p>
<h2 id="section-4">自动刷新</h2>
<p>自动刷新回来了!它使用和 Kibana 其他地方用的面板刷新一样的请求系统,所以,它也可以在各处正常工作,包括 Discover,Visualize 和 Dashboard。</p>
<h2 id="nodejs-">nodejs 后端</h2>
<p>我们把后端实现从 Java(具体地说是 JRuby) 迁移到了更新,更快,兼容性更好的 NodeJS。不要担心,我们会打包好 NodeJS 和 Kibana 在一起,没有 Java 依赖的安装步骤会更简单了。启动命令还是那样: <code class="highlighter-rouge">./bin/kibana</code>,而且启动几乎是即时完成!</p>
<p>另一方面,你需要为你的操作系统选择正确的包下载地址。作为操作系统分发版有区别这个事情的补偿(虽然其实毫不相干),我们免费开放了 SSL 支持功能,不管是从浏览器发出的还是发送去 Elasticsearch 的。</p>
<h2 id="section-5">更多</h2>
<p>好了,牛排上来了开吃。不,其实还没,我们还带来了可配置格式的 CSV 导出,更好的数字处理和一个新的页面风格。谁知道我们还藏了什么呢?或许有?或许没有?唯一的办法就是下载下来你自己找找看;所以,现在就出发吧!一定要抢在别人前面,否则就没你的份了!</p>
<p>最后还是那句话,到 <a href="https://github.com/elasticsearch/kibana">GitHub</a> 上给我们提问题,建议,贡献。或者,如果你跟我们一样喜欢 IRC,加入我们在 Freenode 上的 #kibana 频道。</p>
【翻译】kibana 4 正式就位
2015-02-25T00:00:00+08:00
logstash
kibana
http://chenlinux.com/2015/02/25/kibana-4-literally
<p>原文地址:<a href="http://www.elasticsearch.org/blog/kibana-4-literally/">http://www.elasticsearch.org/blog/kibana-4-literally/</a></p>
<p>Kibana 4 现在,从内到外,从前到后,从唯心到唯物,全方位的,正式达到产品级就绪状态了。好吧,其实一个星期前就准备好了,不过我们希望达到绝对的确保它没问题。现在,我们可以分享这个开心的消息给大家了:Kibana 4.0.0 GA 啦!截图和主要信息见下。如果你也如此激动,我们给你准备好了两步计划:</p>
<ol>
<li>从 <a href="http://www.elasticsearch.org/overview/kibana/installation/">Kibana 4 下载</a>页获取它;</li>
<li>阅读 <a href="http://www.elasticsearch.org/guide/en/kibana/current/index.html">Kibana 4 文档</a> 掌握它。</li>
</ol>
<p>小贴士: 如果你还没准备好,你需要先升级你的集群到 <a href="http://www.elasticsearch.org/downloads/1-4-4/">Elasticsearch 1.4.4</a></p>
<p>小贴士 2: 如果你是从 Kibana4 RC1 升级上来,你需要迁移一下你的配置。<a href="https://gist.github.com/spalger/8daf6c2b7f2954639e38">迁移方式见 gist 链接</a></p>
<p><strong>背后的故事</strong></p>
<p>Kibana 一直是用来解决问题的工具。为什么我每天半夜 2 点要被喊起来?代码什么时候推送到生产环境了?它是不是破坏什么了?嗯,我们解决的就是这些。多年以来,不止一个人被凌晨 2 点喊起来。我知道的,对吧?</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/Screen-Shot-2015-02-17-at-1.25.15-PM-1024x692.png" alt="" /></p>
<p>通常的说,答案越简单的时候,问题其实越难。现在,让我们来解决这个难题,这个问题有三层。解决这个问题,需要分析多个维度,多个字段,多个数据源。Kibana 4 正是我们努力创造来用最短时间和最小的麻烦解决最难的问题的。</p>
<p>我们从 Kibana 3 里学到的东西,都应用到了 Kibana 4 里。为什么满足于在地图上画 1000 个点,而实际上我们可以有一亿个点?为什么满足于一个图上处理一个字段?或者一个面板上一个图?为什么一个仪表板上只能一个索引?让我们生成 5 个场景,跨越 2 个字段对比数据,然后从 3 个索引里读取这些数据,放到一个仪表板里。好,让我们开始,然后就可以吃冰淇淋去了。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/Screen-Shot-2015-02-17-at-1.24.14-PM-1024x624.png" alt="" /></p>
<p><strong>绘图</strong></p>
<p>就像冰淇淋一样,问题也有很多风格。为此,我们把 Kibana 划分成那不勒斯风格,但愿不是你讨厌的风格。如果你是 Kibana 的长期用户,你会在主页的第一个标签 Discover 页上感受到亲切。这页让你快速搜索,查找记录,以解决哪些可以通过单条记录讲清全部故事的简单问题。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/Screen-Shot-2015-02-17-at-1.55.18-PM1-1024x573.png" alt="" /></p>
<p>当事情复杂到简单的搜索无能为力的时候,就需要图表来发挥魔力了。切换到 Visualize 标签,用 Elasticsearch 的聚合来分解数据。Visualize 展开数据的多个维度,让你构建图形、表格、地图,来快速解答哪些你之前从来不知道怎么回答的问题。你首先可能被问到的问题应该是“为什么网站上星期变慢了?”,但是这个问题通过数据显示,其实应该是“为什么圣诞节的时候东京地区的请求平均文件大小陡增了?”</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/Screen-Shot-2015-02-18-at-11.13.37-AM-1024x617.png" alt="" /></p>
<p>最后,把这些合一起放到 Dashboard 上。放到一个大屏幕上然后说:“这是你要的答案,这里有个链接可以以后用。同样,我会写到 wiki 里,把数据导出成 CSV 然后发邮件给你。刚吃了点冰淇淋然后写了我简历的第一节。现在给我送更多的冰淇淋来,我吃完了。”</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/Screen-Shot-2015-02-17-at-3.30.30-PM-1024x715.png" alt="" /></p>
<p>每个标签的细节,请阅读 <a href="http://chenlinux.com/2014/10/07/kibana-4-beta-1-released/">Kibana 4 Beta 1: Released</a> 博文。</p>
<p><strong>后续…</strong></p>
<p>现在可以睡会儿了么?当然不。Kibana 4.1 已经在开发中,我们对未来还有着大计划呢。很多变更在努力让 Kibana 4 更稳定和智能,让我们有一个平台,来构建未来的 Elasticsearch 应用。一切都被设计成可扩展的。比如,可视化部分就可以在它的基础上再构建。开源不仅仅是一个 GitHub 账号,而是我们的一个承诺,让每个人都能在我们的结构上构建创新产品。</p>
<p>阅读<a href="http://www.elasticsearch.org/blog">我们的开发者博客</a>里的文章,构建你自己的 Kibana 可视化,创建你自己的 Elasticsearch 应用。想要先睹为快?看 Spencer Alger 在 <a href="http://www.elasticon.com/">Elastic{ON}15</a> 上的演讲吧。</p>
<p>没有你们就没有我们的现在!所以,还是那句话,到 <a href="https://github.com/elasticsearch/kibana">GitHub</a> 上给我们提问题,建议,贡献。或者,如果你跟我们一样喜欢 IRC,加入我们在 Freenode 上的 #kibana 频道。</p>
<p><strong>额外的话</strong></p>
<p>想了解整个 Kibana 4 故事?阅读之前有关 Kibana 4 beta 的博文:</p>
<ul>
<li><a href="http://chenlinux.com/2014/10/07/kibana-4-beta-1-released/">Kibana 4 Beta 1: Released</a></li>
<li><a href="http://chenlinux.com/2014/11/18/kibana-4-beta-2-get-now/">Kibana 4 Beta 2: Get it now</a></li>
<li><a href="http://chenlinux.com/2014/12/19/kibana-4-beta-3-now-more-filtery/">Kibana 4 Beta 3: Now more filtery</a></li>
<li><a href="http://chenlinux.com/2015/02/25/kibana-4-rc1-is-now-available/">Kibana 4 RC1: Freshly baked</a></li>
</ul>
<p>最后,如果你觉得自己有关于 Kibana 的好故事,我们很乐意倾听。发邮件到 <a href="stories@elasticsearch.com">stories@elasticsearch.com</a> 或者 <a href="http://www.twitter.com/elasticsearch">在 Twitter</a> 上联系,我们会帮你分享成功的喜悦给全世界!</p>
【翻译】用 kibana 4 调查你邻居可能投票给的人
2015-02-25T00:00:00+08:00
logstash
kibana
elasticsearch
http://chenlinux.com/2015/02/25/kibana-4-for-investigating-pacs-super-pacs-and-your-neighbors
<p>原文地址:<a href="http://www.elasticsearch.org/blog/kibana-4-for-investigating-pacs-super-pacs-and-your-neighbors/">http://www.elasticsearch.org/blog/kibana-4-for-investigating-pacs-super-pacs-and-your-neighbors/</a></p>
<p>是时候当一个公众黑客了!我们看到地区和联邦政府每天都公开越来越多的数据以提高行政透明度,包括交通事故,药物不良反应,高校助学金申请,餐厅检查甚至厕所位置都有。现在,所有人都能访问这个数据,分析它,然后构建应用以促进公众利益。公众黑客太棒了!</p>
<p>联邦选举委员会发布了竞选献金数据到它的网站(www.fec.gov)上,包括总统、参议院和众议院的。如同 fec.gov 上所说:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>“In 1975, Congress created the Federal Election Commission (FEC) to administer and enforce the Federal Election Campaign Act (FECA) – the statute that governs the financing of federal elections. The duties of the FEC, which is an independent regulatory agency, are to disclose campaign finance information, to enforce the provisions of the law such as the limits and prohibitions on contributions, and to oversee the public funding of Presidential elections.”
</code></pre>
</div>
<p>向公众提供这些信息是对确保选举过程的完整性是至关重要的。</p>
<p>所以,现在 FEC 提供给了我们原始数据,我们能做什么呢?如果你不认为自己是一个会用 R 分析数据的数据科学家,或者会做漂亮的 D3.js 可视化效果的纽约时报员工,你可能这下就卡住了。不要紧,ELK stack 可以不用多少编程,做到丰富的、可视的,交互式数据分析。数据导入的步骤我会稍后讲,现在,先让我们看看 Kibana 4 能做到些什么。</p>
<h2 id="discover">discover</h2>
<p>Kibana 4 里,你应该从 Discover 标签页开始。这是你得到数据集高阶感观的地方。可以查看实时的数据分布,结构化了的字段列表,一起索引中一些文档的实际内容。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/01_discover-1024x640.png" alt="" /></p>
<p>在上面截图里,我们看到 2013-2014 选举周期里,一共有将近 210 万条个人捐献记录。我们能看到很清晰的捐献记录增加的趋势,以及一些看起来是随机的峰值点。</p>
<p>左侧栏列出了数据集中所有的字段。这提供给我们可以提问的内容。比如,我们现在知道数据里有像姓名、城市、州、捐献数量和捐献日期这些字段,我们就可以构思下面这些问题了:</p>
<ul>
<li>哪个州的捐献数量最大?</li>
<li>哪个州的捐献金额最大?</li>
<li>爱荷华州的个人捐献金额实时变化情况如何?</li>
<li>竞选献金数前 10 名的州里,排名前 3 的城市都是哪些?</li>
<li>我喜欢的明星(比如:格温妮丝·帕特洛)给谁捐款了么?</li>
</ul>
<p>字段列表还能帮助你排除掉一些没法回答的问题。比如,这个记录个人捐献的文件并不包含有关委员会和相关候选人的信息(技术上说,个人捐献的去向是跟候选人相关的)。原始数据里只是记录了委员会和候选人的加密 ID。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/02_committeeID-210x300.png" alt="" /></p>
<p>这样,要问“接收献金最多的 10 个委员会的名字是?”就比较难了。通过 Discover 界面发现这点,有助于引导我们加载额外的数据,丰富这个应用,让它更加有用。</p>
<h2 id="visualize">visualize</h2>
<p>当我们确定了可能要问的一些问题后,我们就可以开始基于数据集的这些属性构建可视化了。以前面说到的一个问题为例。</p>
<p>这是个人献金总额最多的 10 个州的饼图:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/03_piechart1-1024x640.png" alt="" /></p>
<p>看起来没有太多的惊喜,如饼图所示,加利福尼亚,纽约,德克萨斯,佛罗里达,伊利诺斯(美国最大的五个州)贡献了最多的捐赠。华盛顿位列第三是一个有趣的值得调研的问题 - 华盛顿作为州的话应该是倒数第三小的,或许作为联邦政府所在地,更容易引导当地居民参与政治。</p>
<p>饼图很好创建:</p>
<ol>
<li>选择用来确定饼图分片大小的聚合(Aggregation)种类:计数(Count)、总和(Sum)还是去重数(Unique Count)。如果你选择了总和或者去重数,Kibana 还需要知道用哪个字段的值来做这个运算。</li>
<li>选择切片(Split Slices)来切割饼图成片。</li>
<li>选择绘制分片的方式:<br />
a. Aggregation: 选择 “Terms” 因为我们是要基于字段的值来创建分片(“terms” 是 Elasticsearch 里的说法)。<br />
b. Field: 选择要做运算的字段。本例中,我们要按照州来计算献金分布,所以选择 “state”。<br />
c. Order/Size: 选择 “Top” 排序,选择长度为 “10” ,这样就能创建一个前 10 名的饼图。<br />
d. Order by: 本例中你应该是用我们第一步里选过的函数来做排序,不过有些高级场景里你也可以在这里选择其他选项。</li>
<li>点击 Apply 然后你就有一个漂亮的饼图了。</li>
<li>点击右上角的 Save 图标,然后取个名字,这你可以把它添加到 Dashboard 里。</li>
</ol>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/04_saveviz-300x239.png" alt="" /></p>
<p>如果你在数据可视化方面有过一些经验,你可能会想“这家话真是个纯码农。饼图在这种数据分析里就是一个错误的可视化方式。”嗯,你是对的(好吧,希望不包括纯码农部分)。这里使用饼图确实给观众带来一些失真的感观,好像这里面已经包括全部 100% 的数据,就好像加利福尼亚的现金占到全国的四分之一一样。</p>
<p>你可以修改 “size” 参数为 “51”,这样分片数就等于实际的总数。不过如下所示,饼图看起来就不怎么漂亮了:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/05_piechart51-1024x640.png" alt="" /></p>
<p>更好的办法是用另一种可视化方式,比如垂直柱状图(Vertical Bar Chart)。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/06_barchart-1024x640.png" alt="" /></p>
<p>创建垂直柱状图的参数看起来很眼熟。因为这些跟前面创建饼图用过的一模一样,毕竟驱动可视化的背后,实际的请求就是一模一样的。我们只是用一种更不容易被误解的方式来展示而已。</p>
<h2 id="dashboard">dashboard</h2>
<p>创建可视化是蛮有趣的,不过有时候,你更希望把这些合起来放进一个漂亮的仪表板上,在这上面,执行一些聚合分析,通过多维度的字段数据获取有用的结论,然后和别人分享你的发现。</p>
<p>添加可视化到仪表板的时间过程非常直接。你创建好一系列可视化后,在 Dashboard 标签页的右上角点击 Add Visualizatioin 图表,然后开始添加即可!</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/07_addviztodashboard-1024x444.png" alt="" /></p>
<p>小贴士:在你去创建可视化和仪表板之前,最好先约定保存这些元素时采用什么命名规则。比如,统一加上你的 Elasticsearch 索引名或者类型名作为前缀。</p>
<p>然后,你就会有一个像这样的仪表板了:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/08_big_dashboard_ss_final-748x1024.jpg" alt="" /></p>
<h2 id="section">探索</h2>
<p>让我们再看两个潜在的数据场景:一个关注特定的 Super PAC,另一个关注你加血的竞选献金。</p>
<h3 id="pac-">这些 pac 后面都有谁?</h3>
<p>政治行动委员会(Political Action Committees), 或者说 PAC,不是什么新东西了。第一个 PAC 在 1947 年《塔夫脱-哈特利法案》禁止工会和企业花钱影响联邦选举的时候就成立的。</p>
<p>Super PACs 应该是由 2010 年的两个最高法院判决促生的。判决裁定没有捐钱给具体候选人,政党或其他 PAC 的 PAC 组织,可以接收来自个人,公会和企业(包括盈利和非盈利的)的无限额捐款以保证独立的支出。[<a href="http://en.wikipedia.org/wiki/Political_action_committee">http://en.wikipedia.org/wiki/Political_action_committee</a>]</p>
<p>Super PACs 是很多争议和辩论的来源,因为在此之前,竞选献金有很明确的额度限制。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/09_pac01-1024x464.png" alt="" /></p>
<p>在上面截图里,我们看到了一个有关捐献的高层次的师徒。特别是,接收捐献的顶级委员会,委员会类型(比如:Super PAC, PAC,党派等)以及利益集团的类别(比如:公司,公会等)。我可以大概猜出来很多委员会的含义,不过还是有些不太明显 —— 比如 “ACTBLUE” 和 “NEXTGEN CLIMATE ACTION COMMITTEE”。超过 七千七百万美元的献金捐给一个命名模糊不清的委员会,真的是一个值得研究的问题。</p>
<p>你可以在数据表格上点击元素,就能过滤这个数据集了:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/10_pacnextgenclimate-300x247.png" alt="" /></p>
<p>点击 “NEXTGEN CLIMATE ACTION COMMITTEE” 后,Kibana 会刷新所有其他图标,只显示捐献给这个委员会的相关数据。我们立刻就发现了一些有趣的现象:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/11_nextgenclimate_deets-1024x376.png" alt="" /></p>
<p>绝大多数捐献给 “NEXTGEN CLIMATE ACTION COMMITTEE” 的人是:</p>
<ul>
<li>自称职位是“创始人”</li>
<li>雇主为 Fahr, LLC</li>
<li>居住在旧金山</li>
</ul>
<p>你再点击 “FAHR, LLC” 继续钻取,很明显这些献金是来自同一个人:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/12_nextgenclimate_deets2-1024x358.png" alt="" /></p>
<p>在通过雇主下钻之前,我们注意到只有 56 笔献金给 “NEXTGEN CLIMATE ACTION COMMITTEE”。几次点击后,我们发现这个 Super PAC 基本都是从 1 个人以及其他极少数人那获取的资金,我们猜测这群人可能是朋友,同事或者其他关系。</p>
<p>而另一个大型 PAC, “ACTBLUE”,就完全不一样了。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/13_actblue-1024x459.png" alt="" /></p>
<p>给这个 PAC 的捐献非常多(跟上个比是 154448 vs 56),而且捐献来源广泛分布在各个地域:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/14_actblue-geodist-1024x360.png" alt="" /></p>
<p>Elasticsearch 提供的一个更有趣的分析函数是关键词聚合(significant terms aggregation)。你可以在比如欺诈检测、异常检测、推荐等各方面使用关键词。Elasticsearch 官博上有一篇文章介绍这个:<a href="http://www.elasticsearch.org/blog/significant-terms-aggregation/">Significant Terms Aggregation</a>.</p>
<p>对于竞选献金数据集,使用关键词的一个例子就是识别一个特定的查询的统计特征。比如说,在很多 PAC 里,捐献者的职业是律师、退休、法官。所以,对任一 PAC 做职业排行统计,都发现不了什么有价值的信息。而使用关键词聚合,正如在表格中做的,可以看到对于 ActBlue,职位更普遍的应该是教授、自由职业和作家。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/15_actblue-occupations-1024x357.png" alt="" /></p>
<p>我们可以过滤另一个 PAC,民主党全国委员会(Democratic National Committee),会发现这个 PAC 的职位都很常见了:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/16_dnc-occupations-1024x617.png" alt="" /></p>
<p>虽然我们开始的这次探索没有回答出关于这些 PAC 的所有问题,它触发了我希望跟踪的更多问题:</p>
<ul>
<li>谁是 Thomas Steyer ,他跟他的 Super PAC 的另外大概 40 到 50 个捐献者之间是什么关系?</li>
<li>NextGen Climate 和 ActBlue 支持哪个候选人?</li>
<li>这两个组织之间有什么关联?</li>
<li>有没有什么有意无意的帮助特定 PAC 的营销手段,让特定行业的雇员更有兴趣?</li>
</ul>
<p>整个钻取过程的优点是:在帮助回答一些问题的时候,用 ELK stack 还能帮你制定出一些甚至你自己都没想到能问出来的问题!</p>
<h2 id="section-1">我家乡的人把钱给谁了?</h2>
<p>警告:根据你家乡的大小,你可能会发现一些让你邻居很尴尬的事情:)</p>
<p>所有超过 $200 的献金都被要求依法公开,所以,虽然在这里看到你邻居的信息可能比较尴尬,不过竞选县级是公众信息,公众是有这个合法知情权的。</p>
<p>你可以很快的钻取数据集到州、市,然后看到你家乡谁捐献了,捐给了谁。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/17_hoboken-dashboard-1024x640.png" alt="" /></p>
<p>新泽西的霍博肯只有 449 条记录,逐一翻阅记录也花不了多少时间。但是,如果你要分析的是纽约市的 70850 条记录,通过 ELK stack 提供的交互式用户体现就体现出明显优势了:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/18_nyc-dashboard-1024x640.png" alt="" /></p>
<p>回到我的家乡,新泽西的霍博肯,通过几次点击,你就可以构建出为当地参议院和众议院竞选捐献的排行榜。我一直不太明白为什么人们要出钱给 Cory Booker(赢得 56% 选票)和 Albio Sires(赢得 77.3% 选票)参与的非竞争性的比赛。或者只是因为需要支持一下朋友?不过一个关心政治的人,可能就会留意这里面的每一个细节了。</p>
<h2 id="section-2">总结</h2>
<p>我们刚看过了用 ELKstack 探索 FEC 竞选献金数据能做到什么。希望这也能帮你扩展使用 ELKstack 的思路,应用这些数据发现的规则到其他类型的数据是,不管是结构化的比如事务数据,非结构化的比如纯文本数据,抑或二者的混合体。</p>
<p>个人、非营利组织、政府机构和私人公司,从初创公司到大型企业,都在使用 ELK stack 处理实时数据集,大小从几 MB 到几 PB,随着 Kibana 4 的发布,处理会变得更容易和更强大。</p>
<h2 id="a--elk-">附录 a. 如何在笔记本电脑上运行 elk 分析本数据集</h2>
<p>如果你还没有最新版的 ELK stack 的话,可以从 <a href="http://www.elasticsearch.org/overview/elkdownloads/">http://www.elasticsearch.org/overview/elkdownloads/</a> 页面上下载并依照该页说明进行安装。</p>
<p>实际上你并不一定需要 Logstash 来完成这件事情,不过你如果想调试一把 Logstash 配置然后自己加载原始数据,安装 Logstash 还是完全值得的。</p>
<h3 id="elasticsearch-">恢复 elasticsearch 索引镜像</h3>
<p>下载安装完 ELK stack 后,你需要下载献金数据的索引镜像文件(注意:这是一个 1.4GB 大的文件,小心你的手机流量):</p>
<p><a href="http://download.elasticsearch.org/demos/usfec/snapshot_demo_usfec.tar.gz">http://download.elasticsearch.org/demos/usfec/snapshot_demo_usfec.tar.gz</a></p>
<p>在你本地磁盘上创建一个叫 snapshots 的文件夹,然后解压下载的 .tar.gz 文件进去。比如:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>mkdir -p ~/elk/snapshots
cp ~/Downloads/snapshot_demo_usfec.tar.gz ~/elk/snapshots
cd ~/elk/snapshots
tar xf snapshot_demo_usfec.tar.gz
</code></pre>
</div>
<p>等你把 Elasticsearch 跑起来以后,恢复索引就只需要两步了:</p>
<p>1) 为镜像注册一个文件系统仓库(修改下例中 “location” 的值到你实际的 usfec 镜像目录):</p>
<div class="highlighter-rouge"><pre class="highlight"><code>curl -XPUT 'http://localhost:9200/_snapshot/usfec' -d '{
"type": "fs",
"settings": {
"location": "/tmp/snapshots/usfec",
"compress": true,
"max_snapshot_bytes_per_sec": "1000mb",
"max_restore_bytes_per_sec": "1000mb"
}
}'
</code></pre>
</div>
<p>2) 调用恢复接口(Restore API endpoint)开始恢复索引数据到你的 Elasticsearch 实例:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>curl -XPOST "localhost:9200/_snapshot/usfec/1/_restore"
</code></pre>
</div>
<p>现在,去<a href="https://bluebottlecoffee.com/preparation-guides">喝个咖啡</a>。等一会儿后,你可以调用 cat recovery API 来检查一下恢复操作是否完成:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>curl -XGET 'localhost:9200/_cat/recovery?v'
</code></pre>
</div>
<p>或者获取索引的文档数:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>curl -XGET localhost:9200/usfec*/_count -d '{
"query": {
"match_all": {}
}
}'
</code></pre>
</div>
<p>如果全部完成的话,这个数应该是 4250251。</p>
<h3 id="kibana-4--elasticsearch-">指向 kibana 4 到一个 elasticsearch 索引</h3>
<p>你通过 localhost:5601 第一次访问 Kibana 的时候,它会要求你定义一个 “index pattern”:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2010/02/19_indexpattern01-1024x398.png" alt="" /></p>
<p>因为 Elasticsearch 集群可能有多个索引,你需要告诉 Kibana 哪些索引里有你希望读取的数据。在本例中,献金镜像包括了四个索引,当你运行索引恢复操作后,应该在你的 Elasticsearch 实例里创建好了四个新索引:</p>
<ul>
<li>usfec_indiv_contrib: 由个人捐赠给委员会</li>
<li>usfec_comm2cand_contrib: 由委员会捐赠给候选人</li>
<li>usfec_comm2comm_contrib: 由委员会转给其他委员会</li>
<li>usfec_oppexp: 委员会运营支出</li>
</ul>
<p>你可以输入一个索引名字到输入框,然后选择一个时间字段(我们索引里,应该是 <code class="highlighter-rouge">@timestamp</code>),然后点击 Create:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2010/02/20_indexpattern02-1024x487.png" alt="" /></p>
<p>这篇博文的示例中,我们只用到了个人献金的数据,其他三个索引里其实还有很多价值。甚至你可以在 Kibana 里同时指向这四个索引,然后找出不同数据集之间的联系!</p>
<p>打开 Discover 标签,选择一个合适的时间段(选择 “From” 时间为 2012-12-18),开始探索吧!</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2015/02/21_pickdatetimeframe-1024x640.png" alt="" /></p>
<h3 id="b-">附录 b. 参考链接</h3>
<p>fec.gov 的原始数据和数据字典文件<br />
<a href="http://www.fec.gov/finance/disclosure/ftpdet.shtml#a2013_2014">http://www.fec.gov/finance/disclosure/ftpdet.shtml#a2013_2014</a></p>
<p>OpenSecrets.org 资源中心: <br />
分析献金数据的各种资源。感谢这里提供了 FEC 数据更详细的字典。<br />
<a href="https://www.opensecrets.org/resources/create/">https://www.opensecrets.org/resources/create/</a></p>
<p>存放文件的 Github 仓库: <br />
Logstash 配置, 索引模板, 解析数据创建 JSON 的 Python 脚本等<br />
<a href="https://github.com/elasticsearch/demo/tree/master/usfec">https://github.com/elasticsearch/demo/tree/master/usfec</a></p>
spark streaming 的 transform 操作示例
2015-02-14T00:00:00+08:00
monitor
spark
scala
http://chenlinux.com/2015/02/14/spark-streaming-transform
<p>前两篇,一篇说在 spark 里用 SQL 方便,一篇说 updatestateByKey 可以保留状态做推算。那么怎么综合起来呢?目前看到的 spark streaming 和 spark SQL 的示例全都是在 output 阶段的 <code class="highlighter-rouge">foreachRDD</code> 里才调用 SQL。实际在 output 之前,也是可以对 DStream 里的 RDD 做复杂的转换操作的,这就是 <code class="highlighter-rouge">transform</code> 方法。</p>
<p>通过 <code class="highlighter-rouge">transform</code> 方法,可以做到 SQL 请求的结果依然是 DStream 数据,这样就可以使用 <code class="highlighter-rouge">updateStateByKey</code> 方法了。下面是示例:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">org.apache.spark.SparkConf</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.sql._</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.sql.SQLContext</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.SparkContext</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.SparkContext._</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.streaming.</span><span class="o">{</span><span class="n">Seconds</span><span class="o">,</span> <span class="n">StreamingContext</span><span class="o">}</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.streaming.StreamingContext._</span>
<span class="n">object</span> <span class="n">LogStash</span> <span class="o">{</span>
<span class="k">case</span> <span class="kd">class</span> <span class="nc">Status</span><span class="o">(</span><span class="nl">avg:</span><span class="n">Double</span> <span class="o">=</span> <span class="mf">0.0</span><span class="o">,</span> <span class="nl">count:</span><span class="n">Int</span> <span class="o">=</span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
<span class="n">var</span> <span class="n">countTrend</span> <span class="o">=</span> <span class="mf">0.0</span>
<span class="n">var</span> <span class="n">avgTrend</span> <span class="o">=</span> <span class="mf">0.0</span>
<span class="n">def</span> <span class="o">%(</span><span class="nl">prev:</span><span class="n">Status</span><span class="o">):</span> <span class="n">Status</span> <span class="o">=</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">prev</span><span class="o">.</span><span class="na">count</span> <span class="o">></span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">countTrend</span> <span class="o">=</span> <span class="o">(</span><span class="k">this</span><span class="o">.</span><span class="na">count</span> <span class="o">-</span> <span class="n">prev</span><span class="o">.</span><span class="na">count</span><span class="o">).</span><span class="na">toDouble</span> <span class="o">/</span> <span class="n">prev</span><span class="o">.</span><span class="na">count</span>
<span class="o">}</span>
<span class="k">if</span> <span class="o">(</span><span class="n">prev</span><span class="o">.</span><span class="na">avg</span> <span class="o">></span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">avgTrend</span> <span class="o">=</span> <span class="o">(</span><span class="k">this</span><span class="o">.</span><span class="na">avg</span> <span class="o">-</span> <span class="n">prev</span><span class="o">.</span><span class="na">avg</span><span class="o">)</span> <span class="o">/</span> <span class="n">prev</span><span class="o">.</span><span class="na">avg</span>
<span class="o">}</span>
<span class="k">this</span>
<span class="o">}</span>
<span class="n">override</span> <span class="n">def</span> <span class="n">toString</span> <span class="o">=</span> <span class="o">{</span>
<span class="n">s</span><span class="s">"Trend($avg, $count, $avgTrend, $countTrend)"</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">def</span> <span class="n">updatestatefunc</span><span class="o">(</span><span class="nl">newValue:</span> <span class="n">Seq</span><span class="o">[</span><span class="n">Status</span><span class="o">],</span> <span class="nl">oldValue:</span> <span class="n">Option</span><span class="o">[</span><span class="n">Status</span><span class="o">]):</span> <span class="n">Option</span><span class="o">[</span><span class="n">Status</span><span class="o">]</span> <span class="o">=</span> <span class="o">{</span>
<span class="n">val</span> <span class="n">prev</span> <span class="o">=</span> <span class="n">oldValue</span><span class="o">.</span><span class="na">getOrElse</span><span class="o">(</span><span class="n">Status</span><span class="o">())</span>
<span class="n">val</span> <span class="n">current</span> <span class="o">=</span> <span class="k">if</span> <span class="o">(</span><span class="n">newValue</span><span class="o">.</span><span class="na">size</span> <span class="o">></span> <span class="mi">0</span><span class="o">)</span> <span class="n">newValue</span><span class="o">.</span><span class="na">last</span> <span class="o">%</span> <span class="n">prev</span> <span class="k">else</span> <span class="n">Status</span><span class="o">()</span>
<span class="n">Some</span><span class="o">(</span><span class="n">current</span><span class="o">)</span>
<span class="o">}</span>
<span class="n">def</span> <span class="n">main</span><span class="o">(</span><span class="nl">args:</span> <span class="n">Array</span><span class="o">[</span><span class="n">String</span><span class="o">])</span> <span class="o">{</span>
<span class="n">val</span> <span class="n">sparkConf</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SparkConf</span><span class="o">().</span><span class="na">setMaster</span><span class="o">(</span><span class="s">"local[2]"</span><span class="o">).</span><span class="na">setAppName</span><span class="o">(</span><span class="s">"LogStash"</span><span class="o">)</span>
<span class="n">val</span> <span class="n">sc</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SparkContext</span><span class="o">(</span><span class="n">sparkConf</span><span class="o">)</span>
<span class="n">val</span> <span class="n">ssc</span> <span class="o">=</span> <span class="k">new</span> <span class="n">StreamingContext</span><span class="o">(</span><span class="n">sc</span><span class="o">,</span> <span class="n">Seconds</span><span class="o">(</span><span class="mi">10</span><span class="o">))</span>
<span class="n">ssc</span><span class="o">.</span><span class="na">checkpoint</span><span class="o">(</span><span class="s">"/tmp/spark-streaming-logstash"</span><span class="o">)</span>
<span class="n">val</span> <span class="n">sqc</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SQLContext</span><span class="o">(</span><span class="n">sc</span><span class="o">)</span>
<span class="kn">import</span> <span class="nn">sqc._</span>
<span class="n">val</span> <span class="n">lines</span> <span class="o">=</span> <span class="n">ssc</span><span class="o">.</span><span class="na">socketTextStream</span><span class="o">(</span><span class="s">"localhost"</span><span class="o">,</span> <span class="mi">8888</span><span class="o">)</span>
<span class="n">lines</span><span class="o">.</span><span class="na">transform</span><span class="o">(</span> <span class="n">rdd</span> <span class="o">=></span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">rdd</span><span class="o">.</span><span class="na">count</span> <span class="o">></span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
<span class="n">sqc</span><span class="o">.</span><span class="na">jsonRDD</span><span class="o">(</span><span class="n">rdd</span><span class="o">).</span><span class="na">registerTempTable</span><span class="o">(</span><span class="s">"logstash"</span><span class="o">)</span>
<span class="n">val</span> <span class="n">sqlreport</span> <span class="o">=</span> <span class="n">sqc</span><span class="o">.</span><span class="na">sql</span><span class="o">(</span><span class="s">"SELECT message, COUNT(message) AS host_c, AVG(lineno) AS line_a FROM logstash WHERE path = '/var/log/system.log' AND lineno > 70 GROUP BY message ORDER BY host_c DESC LIMIT 100"</span><span class="o">)</span>
<span class="n">sqlreport</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">r</span> <span class="o">=></span> <span class="o">(</span><span class="n">r</span><span class="o">(</span><span class="mi">0</span><span class="o">).</span><span class="na">toString</span> <span class="o">-></span> <span class="n">Status</span><span class="o">(</span><span class="n">r</span><span class="o">(</span><span class="mi">2</span><span class="o">).</span><span class="na">toString</span><span class="o">.</span><span class="na">toDouble</span><span class="o">,</span> <span class="n">r</span><span class="o">(</span><span class="mi">1</span><span class="o">).</span><span class="na">toString</span><span class="o">.</span><span class="na">toInt</span><span class="o">)))</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="n">rdd</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">l</span> <span class="o">=></span> <span class="o">(</span><span class="s">""</span> <span class="o">-></span> <span class="n">Status</span><span class="o">()))</span>
<span class="o">}</span>
<span class="o">}).</span><span class="na">updateStateByKey</span><span class="o">(</span><span class="n">updatestatefunc</span><span class="o">).</span><span class="na">print</span><span class="o">()</span>
<span class="n">ssc</span><span class="o">.</span><span class="na">start</span><span class="o">()</span>
<span class="n">ssc</span><span class="o">.</span><span class="na">awaitTermination</span><span class="o">()</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre>
</div>
<p>这里有一点需要注意,也是耽误我时间最多的地方:<code class="highlighter-rouge">transform</code> 方法的参数和返回,代码里的定义是 <code class="highlighter-rouge">RDD[T]</code> 和 <code class="highlighter-rouge">RDD[U]</code>。我不懂 Java/Scala,以为是只要是 RDD 对象即可。实践证明,其实要任意场合下返回的 RDD 里的数据类型也保持一致。</p>
<p>在上例中,就是 if 条件下返回的是 <code class="highlighter-rouge">RDD[(String, Status)]</code>,那么 else 条件下,也必须返回一个 <code class="highlighter-rouge">RDD[(String, Status)]</code>,如果直接返回原始的 rdd(也就是 <code class="highlighter-rouge">RDD[String]</code>),就会报错。</p>
spark streaming 的 state 操作示例
2015-02-14T00:00:00+08:00
monitor
spark
scala
http://chenlinux.com/2015/02/14/spark-streaming-state
<p>前一篇学习演示了 spark streaming 的基础运用。下一步进入稍微难一点的,利用 checkpoint 来保留上一个窗口的状态,这样可以做到移动窗口的更新统计。</p>
<p>首先还是先演示一下 spark 里传回调函数的用法,上一篇里用 DStream 处理模拟了 <code class="highlighter-rouge">SUM()</code>,这个纯加法是最简单的了,那么如果 <code class="highlighter-rouge">AVG()</code> 怎么做呢?</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">val</span> <span class="n">r</span> <span class="o">=</span> <span class="n">logs</span><span class="o">.</span><span class="na">filter</span><span class="o">(</span><span class="n">l</span> <span class="o">=></span> <span class="n">l</span><span class="o">.</span><span class="na">path</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="s">"/var/log/system.log"</span><span class="o">)).</span><span class="na">filter</span><span class="o">(</span><span class="n">l</span> <span class="o">=></span> <span class="n">l</span><span class="o">.</span><span class="na">lineno</span> <span class="o">></span> <span class="mi">70</span><span class="o">)</span>
<span class="n">r</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">l</span> <span class="o">=></span> <span class="n">l</span><span class="o">.</span><span class="na">message</span> <span class="o">-></span> <span class="o">(</span><span class="n">l</span><span class="o">.</span><span class="na">lineno</span><span class="o">,</span> <span class="mi">1</span><span class="o">)).</span><span class="na">reduceByKey</span><span class="o">((</span><span class="n">a</span><span class="o">,</span> <span class="n">b</span><span class="o">)</span> <span class="o">=></span> <span class="o">{</span>
<span class="o">(</span><span class="n">a</span><span class="o">.</span><span class="na">_1</span> <span class="o">+</span> <span class="n">b</span><span class="o">.</span><span class="na">_1</span><span class="o">,</span> <span class="n">a</span><span class="o">.</span><span class="na">_2</span> <span class="o">+</span> <span class="n">b</span><span class="o">.</span><span class="na">_2</span><span class="o">)</span>
<span class="o">}).</span><span class="na">map</span><span class="o">(</span><span class="n">t</span> <span class="o">=></span> <span class="n">AlertMsg</span><span class="o">(</span><span class="n">t</span><span class="o">.</span><span class="na">_1</span><span class="o">,</span> <span class="n">t</span><span class="o">.</span><span class="na">_2</span><span class="o">.</span><span class="na">_2</span><span class="o">,</span> <span class="n">t</span><span class="o">.</span><span class="na">_2</span><span class="o">.</span><span class="na">_1</span><span class="o">/</span><span class="n">t</span><span class="o">.</span><span class="na">_2</span><span class="o">.</span><span class="na">_2</span><span class="o">)).</span><span class="na">print</span><span class="o">()</span>
</code></pre>
</div>
<p>这段跟之前做 SUM 的那段的区别:</p>
<ol>
<li>DStream 处理成 PairDStream 的时候,Value 不是单纯的 1,而是一个 Seq[Double, Int]。避免了上一个示例里分开两个 DStream 然后再 join 起来的操作;</li>
<li>给 <code class="highlighter-rouge">reduceByKey</code> 传了一个稍微复杂的匿名函数。在这一个函数里计算了 SUM 和 COUNT,后面 map 只需要做一下除法就是 AVG 了。</li>
</ol>
<p>不过这里还用不上上一次窗口的状态。真正需要上一次窗口状态的,是 <code class="highlighter-rouge">reduceByKeyAndWindow</code> 和 <code class="highlighter-rouge">updateStateByKey</code>。<code class="highlighter-rouge">reduceByKeyAndWindow</code> 和 <code class="highlighter-rouge">reduceByKey</code> 的区别,就是除了计算新数据的函数,还要传递一个处理过期数据的函数。</p>
<p>下面用 <code class="highlighter-rouge">updateStateByKey</code> ,演示一下如何计算每个窗口的平均值,跟上一个窗口的平均值的涨跌幅度,如果波动超过 10%,则输出:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">org.apache.spark.SparkConf</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.SparkContext</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.SparkContext._</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.streaming.</span><span class="o">{</span><span class="n">Seconds</span><span class="o">,</span> <span class="n">StreamingContext</span><span class="o">}</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.streaming.StreamingContext._</span>
<span class="kn">import</span> <span class="nn">scala.util.parsing.json.JSON</span>
<span class="n">object</span> <span class="n">LogStash</span> <span class="o">{</span>
<span class="k">case</span> <span class="kd">class</span> <span class="nc">LogStashV1</span><span class="o">(</span><span class="nl">message:</span><span class="n">String</span><span class="o">,</span> <span class="nl">path:</span><span class="n">String</span><span class="o">,</span> <span class="nl">host:</span><span class="n">String</span><span class="o">,</span> <span class="nl">lineno:</span><span class="n">Double</span><span class="o">,</span> <span class="nl">timestamp:</span><span class="n">String</span><span class="o">)</span>
<span class="k">case</span> <span class="kd">class</span> <span class="nc">Status</span><span class="o">(</span><span class="nl">sum:</span><span class="n">Double</span> <span class="o">=</span> <span class="mf">0.0</span><span class="o">,</span> <span class="nl">count:</span><span class="n">Int</span> <span class="o">=</span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
<span class="n">val</span> <span class="n">avg</span> <span class="o">=</span> <span class="n">sum</span> <span class="o">/</span> <span class="n">scala</span><span class="o">.</span><span class="na">math</span><span class="o">.</span><span class="na">max</span><span class="o">(</span><span class="n">count</span><span class="o">,</span> <span class="mi">1</span><span class="o">)</span>
<span class="n">var</span> <span class="n">countTrend</span> <span class="o">=</span> <span class="mf">0.0</span>
<span class="n">var</span> <span class="n">avgTrend</span> <span class="o">=</span> <span class="mf">0.0</span>
<span class="n">def</span> <span class="o">+(</span><span class="nl">sum:</span><span class="n">Double</span><span class="o">,</span> <span class="nl">count:</span><span class="n">Int</span><span class="o">):</span> <span class="n">Status</span> <span class="o">=</span> <span class="o">{</span>
<span class="n">val</span> <span class="n">newStatus</span> <span class="o">=</span> <span class="n">Status</span><span class="o">(</span><span class="n">sum</span><span class="o">,</span> <span class="n">count</span><span class="o">)</span>
<span class="k">if</span> <span class="o">(</span><span class="k">this</span><span class="o">.</span><span class="na">count</span> <span class="o">></span> <span class="mi">0</span> <span class="o">)</span> <span class="o">{</span>
<span class="n">newStatus</span><span class="o">.</span><span class="na">countTrend</span> <span class="o">=</span> <span class="o">(</span><span class="n">count</span> <span class="o">-</span> <span class="k">this</span><span class="o">.</span><span class="na">count</span><span class="o">).</span><span class="na">toDouble</span> <span class="o">/</span> <span class="k">this</span><span class="o">.</span><span class="na">count</span>
<span class="o">}</span>
<span class="k">if</span> <span class="o">(</span><span class="k">this</span><span class="o">.</span><span class="na">avg</span> <span class="o">></span> <span class="mi">0</span> <span class="o">)</span> <span class="o">{</span>
<span class="n">newStatus</span><span class="o">.</span><span class="na">avgTrend</span> <span class="o">=</span> <span class="o">(</span><span class="n">newStatus</span><span class="o">.</span><span class="na">avg</span> <span class="o">-</span> <span class="k">this</span><span class="o">.</span><span class="na">avg</span><span class="o">)</span> <span class="o">/</span> <span class="k">this</span><span class="o">.</span><span class="na">avg</span>
<span class="o">}</span>
<span class="n">newStatus</span>
<span class="o">}</span>
<span class="n">override</span> <span class="n">def</span> <span class="n">toString</span> <span class="o">=</span> <span class="o">{</span>
<span class="n">s</span><span class="s">"Trend($count, $sum, $avg, $countTrend, $avgTrend)"</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">def</span> <span class="n">updatestatefunc</span><span class="o">(</span><span class="nl">newValue:</span> <span class="n">Seq</span><span class="o">[(</span><span class="n">Double</span><span class="o">,</span> <span class="n">Int</span><span class="o">)],</span> <span class="nl">oldValue:</span> <span class="n">Option</span><span class="o">[</span><span class="n">Status</span><span class="o">]):</span> <span class="n">Option</span><span class="o">[</span><span class="n">Status</span><span class="o">]</span> <span class="o">=</span> <span class="o">{</span>
<span class="n">val</span> <span class="n">prev</span> <span class="o">=</span> <span class="n">oldValue</span><span class="o">.</span><span class="na">getOrElse</span><span class="o">(</span><span class="n">Status</span><span class="o">())</span>
<span class="n">var</span> <span class="n">current</span> <span class="o">=</span> <span class="n">prev</span> <span class="o">+</span> <span class="o">(</span> <span class="n">newValue</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">_</span><span class="o">.</span><span class="na">_1</span><span class="o">).</span><span class="na">sum</span><span class="o">,</span> <span class="n">newValue</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">_</span><span class="o">.</span><span class="na">_2</span><span class="o">).</span><span class="na">sum</span> <span class="o">)</span>
<span class="n">Some</span><span class="o">(</span><span class="n">current</span><span class="o">)</span>
<span class="o">}</span>
<span class="n">def</span> <span class="n">main</span><span class="o">(</span><span class="nl">args:</span> <span class="n">Array</span><span class="o">[</span><span class="n">String</span><span class="o">])</span> <span class="o">{</span>
<span class="n">val</span> <span class="n">sparkConf</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SparkConf</span><span class="o">().</span><span class="na">setMaster</span><span class="o">(</span><span class="s">"local[2]"</span><span class="o">).</span><span class="na">setAppName</span><span class="o">(</span><span class="s">"LogStash"</span><span class="o">)</span>
<span class="n">val</span> <span class="n">sc</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SparkContext</span><span class="o">(</span><span class="n">sparkConf</span><span class="o">)</span>
<span class="n">val</span> <span class="n">ssc</span> <span class="o">=</span> <span class="k">new</span> <span class="n">StreamingContext</span><span class="o">(</span><span class="n">sc</span><span class="o">,</span> <span class="n">Seconds</span><span class="o">(</span><span class="mi">10</span><span class="o">))</span>
<span class="n">val</span> <span class="n">lines</span> <span class="o">=</span> <span class="n">ssc</span><span class="o">.</span><span class="na">socketTextStream</span><span class="o">(</span><span class="s">"localhost"</span><span class="o">,</span> <span class="mi">8888</span><span class="o">)</span>
<span class="n">val</span> <span class="n">jsonf</span> <span class="o">=</span> <span class="n">lines</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">JSON</span><span class="o">.</span><span class="na">parseFull</span><span class="o">(</span><span class="n">_</span><span class="o">)).</span><span class="na">map</span><span class="o">(</span><span class="n">_</span><span class="o">.</span><span class="na">get</span><span class="o">.</span><span class="na">asInstanceOf</span><span class="o">[</span><span class="n">scala</span><span class="o">.</span><span class="na">collection</span><span class="o">.</span><span class="na">immutable</span><span class="o">.</span><span class="na">Map</span><span class="o">[</span><span class="n">String</span><span class="o">,</span> <span class="n">Any</span><span class="o">]])</span>
<span class="n">val</span> <span class="n">logs</span> <span class="o">=</span> <span class="n">jsonf</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">data</span> <span class="o">=></span> <span class="n">LogStashV1</span><span class="o">(</span><span class="n">data</span><span class="o">(</span><span class="s">"message"</span><span class="o">).</span><span class="na">toString</span><span class="o">,</span> <span class="n">data</span><span class="o">(</span><span class="s">"path"</span><span class="o">).</span><span class="na">toString</span><span class="o">,</span> <span class="n">data</span><span class="o">(</span><span class="s">"host"</span><span class="o">).</span><span class="na">toString</span><span class="o">,</span> <span class="n">data</span><span class="o">(</span><span class="s">"lineno"</span><span class="o">).</span><span class="na">toString</span><span class="o">.</span><span class="na">toDouble</span><span class="o">,</span> <span class="n">data</span><span class="o">(</span><span class="s">"@timestamp"</span><span class="o">).</span><span class="na">toString</span><span class="o">))</span>
<span class="n">val</span> <span class="n">r</span> <span class="o">=</span> <span class="n">logs</span><span class="o">.</span><span class="na">filter</span><span class="o">(</span><span class="n">l</span> <span class="o">=></span> <span class="n">l</span><span class="o">.</span><span class="na">path</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="s">"/var/log/system.log"</span><span class="o">)).</span><span class="na">filter</span><span class="o">(</span><span class="n">l</span> <span class="o">=></span> <span class="n">l</span><span class="o">.</span><span class="na">lineno</span> <span class="o">></span> <span class="mi">70</span><span class="o">)</span>
<span class="n">r</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">l</span> <span class="o">=></span> <span class="n">l</span><span class="o">.</span><span class="na">message</span> <span class="o">-></span> <span class="o">(</span><span class="n">l</span><span class="o">.</span><span class="na">lineno</span><span class="o">,</span> <span class="mi">1</span><span class="o">)).</span><span class="na">reduceByKey</span><span class="o">((</span><span class="n">a</span><span class="o">,</span> <span class="n">b</span><span class="o">)</span> <span class="o">=></span> <span class="o">{</span>
<span class="o">(</span><span class="n">a</span><span class="o">.</span><span class="na">_1</span> <span class="o">+</span> <span class="n">b</span><span class="o">.</span><span class="na">_1</span><span class="o">,</span> <span class="n">a</span><span class="o">.</span><span class="na">_2</span> <span class="o">+</span> <span class="n">b</span><span class="o">.</span><span class="na">_2</span><span class="o">)</span>
<span class="o">}).</span><span class="na">updateStateByKey</span><span class="o">(</span><span class="n">updatestatefunc</span><span class="o">).</span><span class="na">filter</span><span class="o">(</span><span class="n">t</span> <span class="o">=></span> <span class="n">t</span><span class="o">.</span><span class="na">_2</span><span class="o">.</span><span class="na">avgTrend</span><span class="o">.</span><span class="na">abs</span> <span class="o">></span> <span class="mf">0.1</span><span class="o">).</span><span class="na">print</span><span class="o">()</span>
<span class="n">ssc</span><span class="o">.</span><span class="na">start</span><span class="o">()</span>
<span class="n">ssc</span><span class="o">.</span><span class="na">awaitTermination</span><span class="o">()</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre>
</div>
<p>这里因为流数据只有 sum 和 count,但是又想留存两个 trend 数据,所以使用了一个新的 cast class,把 trend 数据作为 class 的 value member。对于 state 来说,看到的就是一整个 class 了。</p>
<p>依然有参考资料:</p>
<ul>
<li><a href="http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-time-sessionization-with-spark-streaming-and-apache-hadoop/">http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-time-sessionization-with-spark-streaming-and-apache-hadoop/</a></li>
<li><a href="http://www.scottlogic.com/blog/2013/07/29/spark-stream-analysis.html">http://www.scottlogic.com/blog/2013/07/29/spark-stream-analysis.html</a></li>
<li><a href="https://github.com/rshepherd/spark-streaming-average/blob/master/src/main/scala/StreamingAverage.scala">https://github.com/rshepherd/spark-streaming-average/blob/master/src/main/scala/StreamingAverage.scala</a></li>
</ul>
spark streaming 和 spark sql 结合示例
2015-02-13T00:00:00+08:00
monitor
logstash
spark
http://chenlinux.com/2015/02/13/spark-streaming-sql
<p>之前在博客上演示过如果在 spark 里读取 elasticsearch 中的数据。自然往下一步想,是不是可以把一些原先需要定期请求 elasticsearch 的监控内容挪到 spark 里完成?这次就是探讨一下 spark streaming 环境上如何快速统计各维度的数据。期望目标是,可以实现对流数据的异常模式过滤。平常只需要简单调整模式即可。</p>
<h2 id="spark-">spark 基础预备</h2>
<p>之前作为示例,都是直接在 spark-shell 交互式命令行里完成的。这次说说在正式的情况下怎么做。</p>
<p>spark 是用 scala 写的,scala 的打包工具叫 sbt。首先通过 <code class="highlighter-rouge">sudo port install sbt</code> 安装好。然后创建目录:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>mkdir -p ./logstash/src/main/scala/
</code></pre>
</div>
<p>sbt 打包的配置文件则放在 <code class="highlighter-rouge">./logstash/logstash.sbt</code> 位置。内容如下(注意之间的空行是必须的):</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">name</span> <span class="o">:=</span> <span class="s">"LogStash Project"</span>
<span class="n">version</span> <span class="o">:=</span> <span class="s">"1.0"</span>
<span class="n">scalaVersion</span> <span class="o">:=</span> <span class="s">"2.10.4"</span>
<span class="n">libraryDependencies</span> <span class="o">+=</span> <span class="s">"org.apache.spark"</span> <span class="o">%%</span> <span class="s">"spark-core"</span> <span class="o">%</span> <span class="s">"1.2.0"</span>
<span class="n">libraryDependencies</span> <span class="o">+=</span> <span class="s">"org.apache.spark"</span> <span class="o">%%</span> <span class="s">"spark-streaming"</span> <span class="o">%</span> <span class="s">"1.2.0"</span>
<span class="n">libraryDependencies</span> <span class="o">+=</span> <span class="s">"org.apache.spark"</span> <span class="o">%%</span> <span class="s">"spark-sql"</span> <span class="o">%</span> <span class="s">"1.2.0"</span>
</code></pre>
</div>
<p>然后是程序主文件 <code class="highlighter-rouge">./logstash/src/main/scala/LogStash.scala</code>,先来一个最简单的,从 logstash/output/tcp 收数据并解析出来。注意,因为 spark 只能用 pull 方式获取数据,所以 logstash/output/tcp 必须以 <code class="highlighter-rouge">mode => 'server'</code> 方式运行。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>output {
tcp {
codec => json_lines
mode => 'server'
port => 8888
}
}
</code></pre>
</div>
<h2 id="spark-streaming-">spark streaming 基础示例</h2>
<p>编辑主文件如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">org.apache.spark.SparkConf</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.SparkContext</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.SparkContext._</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.streaming.</span><span class="o">{</span><span class="n">Seconds</span><span class="o">,</span> <span class="n">StreamingContext</span><span class="o">}</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.streaming.StreamingContext._</span>
<span class="kn">import</span> <span class="nn">scala.util.parsing.json.JSON</span>
<span class="n">object</span> <span class="n">LogStash</span> <span class="o">{</span>
<span class="n">def</span> <span class="n">main</span><span class="o">(</span><span class="nl">args:</span> <span class="n">Array</span><span class="o">[</span><span class="n">String</span><span class="o">])</span> <span class="o">{</span>
<span class="n">val</span> <span class="n">sparkConf</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SparkConf</span><span class="o">().</span><span class="na">setMaster</span><span class="o">(</span><span class="s">"local[2]"</span><span class="o">).</span><span class="na">setAppName</span><span class="o">(</span><span class="s">"LogStash"</span><span class="o">)</span>
<span class="n">val</span> <span class="n">sc</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SparkContext</span><span class="o">(</span><span class="n">sparkConf</span><span class="o">)</span>
<span class="n">val</span> <span class="n">ssc</span> <span class="o">=</span> <span class="k">new</span> <span class="n">StreamingContext</span><span class="o">(</span><span class="n">sc</span><span class="o">,</span> <span class="n">Seconds</span><span class="o">(</span><span class="mi">10</span><span class="o">))</span>
<span class="n">val</span> <span class="n">lines</span> <span class="o">=</span> <span class="n">ssc</span><span class="o">.</span><span class="na">socketTextStream</span><span class="o">(</span><span class="s">"localhost"</span><span class="o">,</span> <span class="mi">8888</span><span class="o">)</span>
<span class="n">val</span> <span class="n">jsonf</span> <span class="o">=</span> <span class="n">lines</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">JSON</span><span class="o">.</span><span class="na">parseFull</span><span class="o">(</span><span class="n">_</span><span class="o">)).</span><span class="na">map</span><span class="o">(</span><span class="n">_</span><span class="o">.</span><span class="na">get</span><span class="o">.</span><span class="na">asInstanceOf</span><span class="o">[</span><span class="n">scala</span><span class="o">.</span><span class="na">collection</span><span class="o">.</span><span class="na">immutable</span><span class="o">.</span><span class="na">Map</span><span class="o">[</span><span class="n">String</span><span class="o">,</span> <span class="n">Any</span><span class="o">]])</span>
<span class="n">jsonf</span><span class="o">.</span><span class="na">filter</span><span class="o">(</span><span class="n">l</span> <span class="o">=></span> <span class="n">l</span><span class="o">(</span><span class="s">"lineno"</span><span class="o">)==</span><span class="mi">75</span><span class="o">).</span><span class="na">window</span><span class="o">(</span><span class="n">Seconds</span><span class="o">(</span><span class="mi">30</span><span class="o">)).</span><span class="na">foreachRDD</span><span class="o">(</span> <span class="n">rdd</span> <span class="o">=></span> <span class="o">{</span>
<span class="n">rdd</span><span class="o">.</span><span class="na">foreach</span><span class="o">(</span> <span class="n">r</span> <span class="o">=></span> <span class="o">{</span>
<span class="n">println</span><span class="o">(</span><span class="n">r</span><span class="o">(</span><span class="s">"path"</span><span class="o">))</span>
<span class="o">})</span>
<span class="o">})</span>
<span class="n">ssc</span><span class="o">.</span><span class="na">start</span><span class="o">()</span>
<span class="n">ssc</span><span class="o">.</span><span class="na">awaitTermination</span><span class="o">()</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre>
</div>
<p>非常一目了然,每 10 秒挪动一次 window,window 宽度是 30 秒,把 JSON 数据解析出来以后,做过滤和循环输出。这里需要提示一下的是 <code class="highlighter-rouge">.foreachRDD</code> 方法。这是一个 output 方法。spark streaming 里对 input 收到的 DStream 一定要有 output 处理,那么最常见的就是用 foreachRDD 把 DStream 里的 RDDs 循环一遍,做 save 啊,print 啊等等后续。</p>
<p>然后用 sbt 工具编译后就可以运行了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sbt package && ./spark-1.2.0-bin-hadoop2.4/bin/spark-submit --class "LogStash" --master local[2] target/scala-2.10/logstash-project_2.10-1.0.jar
</code></pre>
</div>
<h2 id="sql-">进阶:数据映射和 SQL 处理</h2>
<p>下面看如何在 spark streaming 上使用 spark SQL。前面通过解析 JSON,得到的是 Map 类型的数据,这个无法直接被 SQL 使用。通常的做法是,通过预定的 scala 里的 <code class="highlighter-rouge">cast class</code>,来转换成 spark SQL 支持的表类型。主文件改成这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">org.apache.spark.SparkConf</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.SparkContext</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.SparkContext._</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.streaming.</span><span class="o">{</span><span class="n">Seconds</span><span class="o">,</span> <span class="n">StreamingContext</span><span class="o">}</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.streaming.StreamingContext._</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.sql.SQLContext</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.sql._</span>
<span class="kn">import</span> <span class="nn">scala.util.parsing.json.JSON</span>
<span class="n">object</span> <span class="n">LogStash</span> <span class="o">{</span>
<span class="k">case</span> <span class="kd">class</span> <span class="nc">LogStashV1</span><span class="o">(</span><span class="nl">message:</span><span class="n">String</span><span class="o">,</span> <span class="nl">path:</span><span class="n">String</span><span class="o">,</span> <span class="nl">host:</span><span class="n">String</span><span class="o">,</span> <span class="nl">lineno:</span><span class="n">Double</span><span class="o">,</span> <span class="nl">timestamp:</span><span class="n">String</span><span class="o">)</span>
<span class="k">case</span> <span class="kd">class</span> <span class="nc">AlertMsg</span><span class="o">(</span><span class="nl">host:</span><span class="n">String</span><span class="o">,</span> <span class="nl">count:</span><span class="n">Int</span><span class="o">,</span> <span class="nl">value:</span><span class="n">Double</span><span class="o">)</span>
<span class="n">def</span> <span class="n">main</span><span class="o">(</span><span class="nl">args:</span> <span class="n">Array</span><span class="o">[</span><span class="n">String</span><span class="o">])</span> <span class="o">{</span>
<span class="n">val</span> <span class="n">sparkConf</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SparkConf</span><span class="o">().</span><span class="na">setMaster</span><span class="o">(</span><span class="s">"local[2]"</span><span class="o">).</span><span class="na">setAppName</span><span class="o">(</span><span class="s">"LogStash"</span><span class="o">)</span>
<span class="n">val</span> <span class="n">sc</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SparkContext</span><span class="o">(</span><span class="n">sparkConf</span><span class="o">)</span>
<span class="n">val</span> <span class="n">ssc</span> <span class="o">=</span> <span class="k">new</span> <span class="n">StreamingContext</span><span class="o">(</span><span class="n">sc</span><span class="o">,</span> <span class="n">Seconds</span><span class="o">(</span><span class="mi">10</span><span class="o">))</span>
<span class="n">val</span> <span class="n">sqc</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SQLContext</span><span class="o">(</span><span class="n">sc</span><span class="o">)</span>
<span class="kn">import</span> <span class="nn">sqc._</span>
<span class="n">val</span> <span class="n">lines</span> <span class="o">=</span> <span class="n">ssc</span><span class="o">.</span><span class="na">socketTextStream</span><span class="o">(</span><span class="s">"localhost"</span><span class="o">,</span> <span class="mi">8888</span><span class="o">)</span>
<span class="n">val</span> <span class="n">jsonf</span> <span class="o">=</span> <span class="n">lines</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">JSON</span><span class="o">.</span><span class="na">parseFull</span><span class="o">(</span><span class="n">_</span><span class="o">)).</span><span class="na">map</span><span class="o">(</span><span class="n">_</span><span class="o">.</span><span class="na">get</span><span class="o">.</span><span class="na">asInstanceOf</span><span class="o">[</span><span class="n">scala</span><span class="o">.</span><span class="na">collection</span><span class="o">.</span><span class="na">immutable</span><span class="o">.</span><span class="na">Map</span><span class="o">[</span><span class="n">String</span><span class="o">,</span> <span class="n">Any</span><span class="o">]])</span>
<span class="n">val</span> <span class="n">logs</span> <span class="o">=</span> <span class="n">jsonf</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">data</span> <span class="o">=></span> <span class="n">LogStashV1</span><span class="o">(</span><span class="n">data</span><span class="o">(</span><span class="s">"message"</span><span class="o">).</span><span class="na">toString</span><span class="o">,</span> <span class="n">data</span><span class="o">(</span><span class="s">"path"</span><span class="o">).</span><span class="na">toString</span><span class="o">,</span> <span class="n">data</span><span class="o">(</span><span class="s">"host"</span><span class="o">).</span><span class="na">toString</span><span class="o">,</span> <span class="n">data</span><span class="o">(</span><span class="s">"lineno"</span><span class="o">).</span><span class="na">toString</span><span class="o">.</span><span class="na">toDouble</span><span class="o">,</span> <span class="n">data</span><span class="o">(</span><span class="s">"@timestamp"</span><span class="o">).</span><span class="na">toString</span><span class="o">))</span>
<span class="n">logs</span><span class="o">.</span><span class="na">foreachRDD</span><span class="o">(</span> <span class="n">rdd</span> <span class="o">=></span> <span class="o">{</span>
<span class="n">rdd</span><span class="o">.</span><span class="na">registerAsTable</span><span class="o">(</span><span class="s">"logstash"</span><span class="o">)</span>
<span class="n">val</span> <span class="n">sqlreport</span> <span class="o">=</span> <span class="n">sqc</span><span class="o">.</span><span class="na">sql</span><span class="o">(</span><span class="s">"SELECT message, COUNT(message) AS host_c, SUM(lineno) AS line_a FROM logstash WHERE path = '/var/log/system.log' AND lineno > 70 GROUP BY message ORDER BY host_c DESC LIMIT 100"</span><span class="o">)</span>
<span class="n">sqlreport</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">t</span> <span class="o">=></span> <span class="n">AlertMsg</span><span class="o">(</span><span class="n">t</span><span class="o">(</span><span class="mi">0</span><span class="o">).</span><span class="na">toString</span><span class="o">,</span> <span class="n">t</span><span class="o">(</span><span class="mi">1</span><span class="o">).</span><span class="na">toString</span><span class="o">.</span><span class="na">toInt</span><span class="o">,</span> <span class="n">t</span><span class="o">(</span><span class="mi">2</span><span class="o">).</span><span class="na">toString</span><span class="o">.</span><span class="na">toDouble</span><span class="o">)).</span><span class="na">collect</span><span class="o">().</span><span class="na">foreach</span><span class="o">(</span><span class="n">println</span><span class="o">)</span>
<span class="o">})</span>
<span class="n">ssc</span><span class="o">.</span><span class="na">start</span><span class="o">()</span>
<span class="n">ssc</span><span class="o">.</span><span class="na">awaitTermination</span><span class="o">()</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre>
</div>
<p>通过加载 SQLContext,就可以把 RDD 转换成 table,然后通过 SQL 方式写请求了。这里有一个地方需要注意的是,因为最开始转换 JSON 的时候,键值对的 value 类型是 Any(因为要兼容复杂结构),所以后面赋值的时候需要具体转换成合适的类型。于是悲催的就有了 <code class="highlighter-rouge">.toString.toInt</code> 这样的写法。。。</p>
<h2 id="sql--1">同样效果的非 SQL 实现</h2>
<p>不用 spark SQL 当然也能做到,而且如果需要复杂处理的时候,还少不了自己写。如果把上例中那段 foreachRDD 替换成下面这样,效果是完全一样的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">val</span> <span class="n">r</span> <span class="o">=</span> <span class="n">logs</span><span class="o">.</span><span class="na">filter</span><span class="o">(</span><span class="n">l</span> <span class="o">=></span> <span class="n">l</span><span class="o">.</span><span class="na">path</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="s">"/var/log/system.log"</span><span class="o">)).</span><span class="na">filter</span><span class="o">(</span><span class="n">l</span> <span class="o">=></span> <span class="n">l</span><span class="o">.</span><span class="na">lineno</span> <span class="o">></span> <span class="mi">70</span><span class="o">)</span>
<span class="n">val</span> <span class="n">host_c</span> <span class="o">=</span> <span class="n">r</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">l</span> <span class="o">=></span> <span class="n">l</span><span class="o">.</span><span class="na">message</span> <span class="o">-></span> <span class="mi">1</span><span class="o">).</span><span class="na">reduceByKey</span><span class="o">(</span><span class="n">_</span><span class="o">+</span><span class="n">_</span><span class="o">).</span><span class="na">groupByKey</span><span class="o">()</span>
<span class="n">r</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">l</span> <span class="o">=></span> <span class="n">l</span><span class="o">.</span><span class="na">message</span> <span class="o">-></span> <span class="n">l</span><span class="o">.</span><span class="na">lineno</span><span class="o">).</span><span class="na">reduceByKey</span><span class="o">(</span><span class="n">_</span><span class="o">+</span><span class="n">_</span><span class="o">).</span><span class="na">groupByKey</span><span class="o">().</span><span class="na">join</span><span class="o">(</span><span class="n">host_c</span><span class="o">).</span><span class="na">foreachRDD</span><span class="o">(</span> <span class="n">rdd</span> <span class="o">=></span> <span class="o">{</span>
<span class="n">rdd</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">t</span> <span class="o">=></span> <span class="n">AlertMsg</span><span class="o">(</span><span class="n">t</span><span class="o">.</span><span class="na">_1</span><span class="o">,</span> <span class="n">t</span><span class="o">.</span><span class="na">_2</span><span class="o">.</span><span class="na">_2</span><span class="o">.</span><span class="na">head</span><span class="o">,</span> <span class="n">t</span><span class="o">.</span><span class="na">_2</span><span class="o">.</span><span class="na">_1</span><span class="o">.</span><span class="na">head</span><span class="o">)).</span><span class="na">collect</span><span class="o">().</span><span class="na">foreach</span><span class="o">(</span><span class="n">println</span><span class="o">)</span>
<span class="o">})</span>
</code></pre>
</div>
<p>这里面用到的 <code class="highlighter-rouge">.groupByKey</code> 和 <code class="highlighter-rouge">.reduceByKey</code> 方法,都是专门针对 PairsDStream 对象的,所以前面必须通过 <code class="highlighter-rouge">.map</code> 方法把普通 DStream 转换一下。</p>
<p>这里还有一个很厉害的方法,叫 <code class="highlighter-rouge">.updatestateByKey</code> 。可以有一个 checkpoint 存上一个 window 的数据,具体示例稍后更新。</p>
<h2 id="jsonrdd-">更简洁的 jsonRDD 方法</h2>
<p>在简单需求的时候,可能还是觉得能用 SQL 就用 SQL 比较好。但是提前定义 cast class 真的比较麻烦。其实对于 JSON 数据,spark SQL 是有提供更简洁的处理接口的。可以直接写成这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">org.apache.spark.SparkConf</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.SparkContext</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.SparkContext._</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.streaming.</span><span class="o">{</span><span class="n">Seconds</span><span class="o">,</span> <span class="n">StreamingContext</span><span class="o">}</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.streaming.StreamingContext._</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.sql.SQLContext</span>
<span class="kn">import</span> <span class="nn">org.apache.spark.sql._</span>
<span class="n">object</span> <span class="n">LogStash</span> <span class="o">{</span>
<span class="k">case</span> <span class="kd">class</span> <span class="nc">AlertMsg</span><span class="o">(</span><span class="nl">host:</span><span class="n">String</span><span class="o">,</span> <span class="nl">count:</span><span class="n">String</span><span class="o">,</span> <span class="nl">value:</span><span class="n">String</span><span class="o">)</span>
<span class="n">def</span> <span class="n">main</span><span class="o">(</span><span class="nl">args:</span> <span class="n">Array</span><span class="o">[</span><span class="n">String</span><span class="o">])</span> <span class="o">{</span>
<span class="n">val</span> <span class="n">sparkConf</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SparkConf</span><span class="o">().</span><span class="na">setMaster</span><span class="o">(</span><span class="s">"local[2]"</span><span class="o">).</span><span class="na">setAppName</span><span class="o">(</span><span class="s">"LogStash"</span><span class="o">)</span>
<span class="n">val</span> <span class="n">sc</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SparkContext</span><span class="o">(</span><span class="n">sparkConf</span><span class="o">)</span>
<span class="n">val</span> <span class="n">ssc</span> <span class="o">=</span> <span class="k">new</span> <span class="n">StreamingContext</span><span class="o">(</span><span class="n">sc</span><span class="o">,</span> <span class="n">Seconds</span><span class="o">(</span><span class="mi">10</span><span class="o">))</span>
<span class="n">val</span> <span class="n">sqc</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SQLContext</span><span class="o">(</span><span class="n">sc</span><span class="o">)</span>
<span class="kn">import</span> <span class="nn">sqc._</span>
<span class="n">val</span> <span class="n">lines</span> <span class="o">=</span> <span class="n">ssc</span><span class="o">.</span><span class="na">socketTextStream</span><span class="o">(</span><span class="s">"localhost"</span><span class="o">,</span> <span class="mi">8888</span><span class="o">)</span>
<span class="n">lines</span><span class="o">.</span><span class="na">foreachRDD</span><span class="o">(</span> <span class="n">rdd</span> <span class="o">=></span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">rdd</span><span class="o">.</span><span class="na">count</span> <span class="o">></span> <span class="mi">0</span><span class="o">)</span> <span class="o">{</span>
<span class="n">val</span> <span class="n">t</span> <span class="o">=</span> <span class="n">sqc</span><span class="o">.</span><span class="na">jsonRDD</span><span class="o">(</span><span class="n">rdd</span><span class="o">)</span>
<span class="c1">// t.printSchema()</span>
<span class="n">t</span><span class="o">.</span><span class="na">registerTempTable</span><span class="o">(</span><span class="s">"logstash"</span><span class="o">)</span>
<span class="n">val</span> <span class="n">sqlreport</span> <span class="o">=</span><span class="n">sqc</span><span class="o">.</span><span class="na">sql</span><span class="o">(</span><span class="s">"SELECT host, COUNT(host) AS host_c, AVG(lineno) AS line_a FROM logstash WHERE path = '/var/log/system.log' AND lineno > 70 GROUP BY host ORDER BY host_c DESC LIMIT 100"</span><span class="o">)</span>
<span class="n">sqlreport</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">t</span><span class="o">=></span> <span class="n">AlertMsg</span><span class="o">(</span><span class="n">t</span><span class="o">(</span><span class="mi">0</span><span class="o">).</span><span class="na">toString</span><span class="o">,</span><span class="n">t</span><span class="o">(</span><span class="mi">1</span><span class="o">).</span><span class="na">toString</span><span class="o">,</span><span class="n">t</span><span class="o">(</span><span class="mi">2</span><span class="o">).</span><span class="na">toString</span><span class="o">)).</span><span class="na">collect</span><span class="o">().</span><span class="na">foreach</span><span class="o">(</span><span class="n">println</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">})</span>
<span class="n">ssc</span><span class="o">.</span><span class="na">start</span><span class="o">()</span>
<span class="n">ssc</span><span class="o">.</span><span class="na">awaitTermination</span><span class="o">()</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre>
</div>
<p>这样,不用自己解析 JSON,直接加载到 SQLContext 里。可以通过 <code class="highlighter-rouge">.printSchema</code> 方法查看到 JSON 被转换成了什么样的表结构。</p>
<h2 id="todo">TODO</h2>
<p>SQL 的方式可以很方便的做到对实时数据的阈值监控处理,但是 SQL 是建立在 RDD 上的如何利用 DStream 的上一个 window 的 state 状态实现比如环比变化处理,移动均线处理,还没找到途径。</p>
<h2 id="see-also">See Also</h2>
<p>spark 目前文档不多,尤其是 streaming 和 SQL 方面的。感谢下面两个网址,对我上手帮助颇多:</p>
<ul>
<li><a href="http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Json-file-groupby-function-td9618.html">http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Json-file-groupby-function-td9618.html</a></li>
<li><a href="http://databricks.gitbooks.io/databricks-spark-reference-applications/content/logs_analyzer/chapter1/windows.html">http://databricks.gitbooks.io/databricks-spark-reference-applications/content/logs_analyzer/chapter1/windows.html</a></li>
</ul>
rsyslog 的 TCP 转发性能测试
2015-02-12T00:00:00+08:00
performance
rsyslog
http://chenlinux.com/2015/02/12/rsyslog-forwarder-testing
<p>做一个日志手机系统,一般有两个思路。一个是提供一个多语言 SDK 包,然后开发者只需要找到对应的 SDK 加载即可;一个是采用最通用的日志传输协议,让开发者采用现成的协议实现。在通用协议里,最常见的,就是 syslog 协议。不过 syslog 过去采用 UDP 的印象太过深入人心,rsyslog 虽然宣称在测试用达到了每秒上百万的性能,也没多少人相信。那么,到底用 syslog 协议做跨网络传输,靠不靠谱?自己用压测,来证明一下!</p>
<h2 id="section">测试环境</h2>
<p>两台测试机。其中:</p>
<p>A 配置为 imtcp/514,omfwd 到 B 的 514。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>module( load="imtcp" )
input( type="imtcp" port="514" ruleset="forwardruleset" )
Ruleset( name="forwardruleset" )
{
action (
type="omfwd"
Target="$b-server-ip"
Port="514"
Protocol="tcp"
RebindInterval="5000"
name="action_fwd"
queue.filename="action_fwd"
queue.size="50000"
queue.dequeuebatchsize="1000"
queue.maxdiskspace="5G"
queue.discardseverity="3"
queue.checkpointinterval="10"
queue.type="linkedlist"
queue.workerthreads="1"
queue.timeoutshutdown="10"
queue.timeoutactioncompletion="10"
queue.timeoutenqueue="20"
queue.timeoutworkerthreadshutdown="10"
queue.workerthreadminimummessages="5000"
queue.maxfilesize="500M"
queue.saveonshutdown="on"
)
stop
}
</code></pre>
</div>
<p>B 配置为 imtcp/514,omfile 到本机。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>module( load="imtcp" )
input( type="imtcp" port="514" ruleset="recordruleset" )
Ruleset( name="recordruleset" )
{
action( type="omfile" file="/data1/debug.log" template="defaultLogFormat" asyncWriting="on" flushOnTXEnd="off" ioBufferSize="81920k" flushInterval="5")
}
</code></pre>
</div>
<h2 id="section-1">测试工具</h2>
<p>为了控制测试的速度,放弃之前压测 logstash 时候用的 logger 命令,采用 syslog-ng 项目自带的 loggen 命令。本来准备编译一下 syslog-ng,不过报错太多,实在复杂,看了一下 loggen.c 本身没啥依赖,所以决定采用最简单的办法获取 loggen 命令——下载 syslog-ng.rpm,然后直接解压压缩包!</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget http://mirrors.zju.edu.cn/epel/5/x86_64/syslog-ng-2.1.4-9.el5.x86_64.rpm
rpm2cpio syslog-ng-2.1.4-9.el5.x86_64.rpm | cpio -div
</code></pre>
</div>
<p><em>我这不能直接通过 yum install 安装,因为 syslog-ng 跟系统里已有的 rsyslog 是冲突的。</em></p>
<h2 id="section-2">测试命令</h2>
<p>rpm 获取的 loggen 命令还不支持 <code class="highlighter-rouge">--read-data</code> 参数,只能自己模拟填充数据。所以测试命令如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>./usr/bin/loggen -r 10000 -i -s 500 -I 600 $a-server-ip 514
</code></pre>
</div>
<p>意即单条长度 500 字节,每秒 10000 条的频率,持续发送 600 秒。</p>
<h2 id="section-3">验证方式</h2>
<p>rsyslog 有专门的 impstats 模块,输出本身运行情况的统计,可以通过如下配置开启:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>module( load="impstats" interval="60" severity="6" log.syslog="on" format="json" resetCounters="on")
template( name="dynaFileRsyslog" type="string" string="/data1/rsyslog/impstats/%$year%/%$month%/%$day%_impstats.log" )
if ( $syslogfacility-text == 'syslog' ) then
{
action ( type="omfile" DynaFile="dynaFileRsyslog" FileCreateMode="0600" )
stop
}
</code></pre>
</div>
<p>每 60 秒会输出 JSON 格式的统计数据,类似这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>2015-02-11T20:00:43.176325+08:00 localhost rsyslogd-pstats: {"name":"action_fwd queue","size":0,"enqueued":0,"full":0,"discarded.full":0,"discarded.nf":0,"maxqsize":0}
</code></pre>
</div>
<p>其中,enqueued 表示进入队列的条目数,size 表示暂存在内存中的条目数,discarded.full 表示队列满丢弃的条目数,discarded.nf 表示队列将满丢弃的条目数。</p>
<p>如果内存队列都不够用,那么 rsyslog 会记录到磁盘队列上,这时候看到类似上面的统计数据的另一条记录,区别是 <code class="highlighter-rouge">"name":"action_fwd queue[DA]"</code>,这个 DA 就是磁盘队列的意思。</p>
<h2 id="section-4">测试结果</h2>
<ol>
<li>每秒 5 万条的发送,可以做到毫无 size 的全部即时转发。</li>
<li>加大 <code class="highlighter-rouge">queue.size</code> 到 10 倍,即时转发能力提高到 12 万条。</li>
<li>再加大 <code class="highlighter-rouge">queue.workerthreads</code> 到 10,即时转发能力提高到 15 万条。</li>
<li>单独加大 <code class="highlighter-rouge">queue.dequeuebatchsize</code> 到 10 倍,即时转发能力提高到 17 万条。</li>
<li>同时加大 <code class="highlighter-rouge">queue.size</code> 和 <code class="highlighter-rouge">queue.dequeuebatchsize</code> 到 10 倍 ,即时转发能力提高到 18 万条。</li>
<li>加大频率到 24 万,进入磁盘队列,因为这时候已经到千兆网卡瓶颈。</li>
<li>加大模拟长度到 5000 字节,即时转发能力下降到 1 万。</li>
</ol>
<p>最后,尽可能删除掉各种配置,以默认方式运行,发现转发能力也能达到 5 万条。查了一下源码,默认的 <code class="highlighter-rouge">queue.size</code> 是 1000,<code class="highlighter-rouge">queue.dequeuebatchsize</code> 是 16。说明在这段大小(初始测试值是默认值的 50 多倍)内,性能变化不大。</p>
<h2 id="section-5">长期运行</h2>
<p>测试每次只运行几分钟,还需要长期运行的考验。运行两三天的观察,同时加大到 10 倍的配置(即短期测试可以跑满网卡的配置),在长期稳定每秒 5 万条的测试中,也会出现内存队列的 size 数。还需继续观察 size 是否累积,以及更大量的情况是否会出现磁盘队列。</p>
Python 批量写入 Elasticsearch 脚本
2015-02-11T00:00:00+08:00
logstash
python
pypy
elasticsearch
http://chenlinux.com/2015/02/11/python-elasticsearch-bulk
<p>Elasticsearch 官方和社区提供了各种各样的客户端库,在之前的博客中,我陆陆续续提到和演示过 Perl 的,Javascript 的,Ruby 的。上周写了一版 Python 的,考虑到好像很难找到现成的示例,如何用 python 批量写数据进 Elasticsearch,今天一并贴上来。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#!/usr/bin/env pypy</span>
<span class="c">#coding:utf-8</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">datetime</span>
<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">from</span> <span class="nn">elasticsearch</span> <span class="kn">import</span> <span class="n">Elasticsearch</span>
<span class="kn">from</span> <span class="nn">elasticsearch</span> <span class="kn">import</span> <span class="n">helpers</span>
<span class="kn">from</span> <span class="nn">elasticsearch</span> <span class="kn">import</span> <span class="n">ConnectionTimeout</span>
<span class="n">es</span> <span class="o">=</span> <span class="n">Elasticsearch</span><span class="p">([</span><span class="s">'192.168.0.2'</span><span class="p">,</span> <span class="s">'192.168.0.3'</span><span class="p">],</span> <span class="n">sniff_on_start</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">sniff_on_connection_fail</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">max_retries</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">retry_on_timeout</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">()</span>
<span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s">'elasticsearch'</span><span class="p">)</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">WARN</span><span class="p">)</span>
<span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s">'urllib3'</span><span class="p">)</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">WARN</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">parse_www</span><span class="p">(</span><span class="n">logline</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">time_local</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">http_user_agent</span><span class="p">,</span> <span class="n">staTus</span><span class="p">,</span> <span class="n">remote_addr</span><span class="p">,</span> <span class="n">http_referer</span><span class="p">,</span> <span class="n">request_time</span><span class="p">,</span> <span class="n">body_bytes_sent</span><span class="p">,</span> <span class="n">http_x_forwarded_proto</span><span class="p">,</span> <span class="n">http_x_forwarded_for</span><span class="p">,</span> <span class="n">http_host</span><span class="p">,</span> <span class="n">http_cookie</span><span class="p">,</span> <span class="n">upstream_response_time</span> <span class="o">=</span> <span class="n">logline</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">'`'</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">upstream_response_time</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="n">upstream_response_time</span><span class="p">)</span>
<span class="k">except</span><span class="p">:</span>
<span class="n">upstream_response_time</span> <span class="o">=</span> <span class="bp">None</span>
<span class="n">method</span><span class="p">,</span> <span class="n">uri</span><span class="p">,</span> <span class="n">verb</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">' '</span><span class="p">)</span>
<span class="n">arg</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">url_path</span><span class="p">,</span> <span class="n">url_args</span> <span class="o">=</span> <span class="n">uri</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">'?'</span><span class="p">)</span>
<span class="k">for</span> <span class="n">args</span> <span class="ow">in</span> <span class="n">url_args</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">'&'</span><span class="p">):</span>
<span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="o">=</span> <span class="n">args</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">'='</span><span class="p">)</span>
<span class="n">arg</span><span class="p">[</span><span class="n">k</span><span class="p">]</span> <span class="o">=</span> <span class="n">v</span>
<span class="k">except</span><span class="p">:</span>
<span class="n">url_path</span> <span class="o">=</span> <span class="n">uri</span>
<span class="c"># Why %z do not implement?</span>
<span class="n">date</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">time_local</span><span class="p">,</span> <span class="s">'[</span><span class="si">%</span><span class="s">d/</span><span class="si">%</span><span class="s">b/</span><span class="si">%</span><span class="s">Y:</span><span class="si">%</span><span class="s">H:</span><span class="si">%</span><span class="s">M:</span><span class="si">%</span><span class="s">S +0800]'</span><span class="p">)</span>
<span class="n">ret</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">"@timestamp"</span><span class="p">:</span> <span class="n">date</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s">'</span><span class="si">%</span><span class="s">FT</span><span class="si">%</span><span class="s">T+0800'</span><span class="p">),</span>
<span class="s">"host"</span><span class="p">:</span> <span class="s">"127.0.0.1"</span><span class="p">,</span>
<span class="s">"method"</span><span class="p">:</span> <span class="n">method</span><span class="o">.</span><span class="n">lstrip</span><span class="p">(</span><span class="s">'"'</span><span class="p">),</span>
<span class="s">"url_path"</span><span class="p">:</span> <span class="n">url_path</span><span class="p">,</span>
<span class="s">"url_args"</span><span class="p">:</span> <span class="n">arg</span><span class="p">,</span>
<span class="s">"verb"</span><span class="p">:</span> <span class="n">verb</span><span class="o">.</span><span class="n">rstrip</span><span class="p">(</span><span class="s">'"'</span><span class="p">),</span>
<span class="s">"http_user_agent"</span><span class="p">:</span> <span class="n">http_user_agent</span><span class="p">,</span>
<span class="s">"status"</span><span class="p">:</span> <span class="nb">int</span><span class="p">(</span><span class="n">staTus</span><span class="p">),</span>
<span class="s">"remote_addr"</span><span class="p">:</span> <span class="n">remote_addr</span><span class="o">.</span><span class="n">strip</span><span class="p">(</span><span class="s">'[]'</span><span class="p">),</span>
<span class="s">"http_referer"</span><span class="p">:</span> <span class="n">http_referer</span><span class="p">,</span>
<span class="s">"request_time"</span><span class="p">:</span> <span class="nb">float</span><span class="p">(</span><span class="n">request_time</span><span class="p">),</span>
<span class="s">"body_bytes_sent"</span><span class="p">:</span> <span class="nb">int</span><span class="p">(</span><span class="n">body_bytes_sent</span><span class="p">),</span>
<span class="s">"http_x_forwarded_proto"</span><span class="p">:</span> <span class="n">http_x_forwarded_proto</span><span class="p">,</span>
<span class="s">"http_x_forwarded_for"</span><span class="p">:</span> <span class="n">http_x_forwarded_for</span><span class="p">,</span>
<span class="s">"http_host"</span><span class="p">:</span> <span class="n">http_host</span><span class="p">,</span>
<span class="s">"http_cookie"</span><span class="p">:</span> <span class="n">http_cookie</span><span class="p">,</span>
<span class="s">"upstream_response_time"</span><span class="p">:</span> <span class="n">upstream_response_time</span>
<span class="p">}</span>
<span class="k">return</span> <span class="p">{</span><span class="s">"_index"</span><span class="p">:</span><span class="s">"logstash-mweibo-www-"</span><span class="o">+</span><span class="n">date</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s">'</span><span class="si">%</span><span class="s">Y.</span><span class="si">%</span><span class="s">m.</span><span class="si">%</span><span class="s">d'</span><span class="p">),</span> <span class="s">"_type"</span><span class="p">:</span><span class="s">"nginx"</span><span class="p">,</span><span class="s">"_source"</span><span class="p">:</span><span class="n">ret</span><span class="p">}</span>
<span class="k">except</span><span class="p">:</span>
<span class="k">return</span> <span class="p">{</span><span class="s">"_index"</span><span class="p">:</span><span class="s">"logstash-mweibo-www-"</span><span class="o">+</span><span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s">'</span><span class="si">%</span><span class="s">Y.</span><span class="si">%</span><span class="s">m.</span><span class="si">%</span><span class="s">d'</span><span class="p">),</span> <span class="s">"_type"</span><span class="p">:</span><span class="s">"nginx"</span><span class="p">,</span><span class="s">"_source"</span><span class="p">:{</span><span class="s">"message"</span><span class="p">:</span><span class="n">logline</span><span class="p">}}</span>
<span class="k">def</span> <span class="nf">get_log</span><span class="p">():</span>
<span class="n">start_time</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
<span class="n">log_buffer</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">line</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span>
<span class="k">except</span><span class="p">:</span>
<span class="k">break</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">line</span><span class="p">:</span>
<span class="n">helpers</span><span class="o">.</span><span class="n">bulk</span><span class="p">(</span><span class="n">es</span><span class="p">,</span> <span class="n">log_buffer</span><span class="p">)</span>
<span class="k">del</span> <span class="n">log_buffer</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="nb">len</span><span class="p">(</span><span class="n">log_buffer</span><span class="p">)]</span>
<span class="k">break</span>
<span class="k">if</span> <span class="n">line</span><span class="p">:</span>
<span class="n">ret</span> <span class="o">=</span> <span class="n">parse_www</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">rstrip</span><span class="p">())</span>
<span class="n">log_buffer</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">ret</span><span class="p">)</span>
<span class="k">while</span> <span class="p">(</span> <span class="nb">len</span><span class="p">(</span><span class="n">log_buffer</span><span class="p">)</span> <span class="o">></span> <span class="mi">2000</span> <span class="ow">and</span> <span class="nb">len</span><span class="p">(</span><span class="n">log_buffer</span><span class="p">)</span> <span class="o">%</span> <span class="mi">2000</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">helpers</span><span class="o">.</span><span class="n">bulk</span><span class="p">(</span><span class="n">es</span><span class="p">,</span> <span class="n">log_buffer</span><span class="p">)</span>
<span class="k">except</span> <span class="n">ConnectionTimeout</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"try again"</span><span class="p">)</span>
<span class="k">continue</span>
<span class="k">del</span> <span class="n">log_buffer</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="nb">len</span><span class="p">(</span><span class="n">log_buffer</span><span class="p">)]</span>
<span class="k">break</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">if</span> <span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">-</span> <span class="n">startime</span> <span class="o">></span> <span class="n">timeout</span> <span class="p">):</span>
<span class="n">helpers</span><span class="o">.</span><span class="n">bulk</span><span class="p">(</span><span class="n">es</span><span class="p">,</span> <span class="n">log_buffer</span><span class="p">)</span>
<span class="n">start_time</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
<span class="k">del</span> <span class="n">log_buffer</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="nb">len</span><span class="p">(</span><span class="n">log_buffer</span><span class="p">)]</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">get_log</span><span class="p">()</span>
</code></pre>
</div>
<p>和 Perl、Ruby 的客户端不同,Python 的客户端只支持两种 transport 方式,urllib3 或者 thrift。也就是说,木有像事件驱动啊之类的办法。</p>
<p>测试一下,这个脚本如果不发送数据,一秒处理日志条数在15k,发送数据,一秒只有2k。确实比较让人失望,于是决定换成 pypy 试试——我司不少日志处理脚本都是用 pypy 运行的。</p>
<p>服务器上使用 pypy ,是通过 EPEL 安装的,之前都只用核心模块,这次需要安装 elasticsearch 模块。所以需要先给 pypy 加上 pip:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py
pypy get-pip.py
</code></pre>
</div>
<p>网上大多说之前还要下载一个叫 distribute_setup.py 的脚本来运行,实测不需要,而且这个脚本的下载链接也失效了。</p>
<p>然后通过 pip 安装 elasticsearch 包即可:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>/usr/lib64/pypy-2.0.2/bin/pip install elasticsearch
</code></pre>
</div>
<p>测试,pypy 比 python 处理日志速度快一倍,写 ES 速度快一半。不过 3300eps 依然很慢就是了。</p>
<h2 id="section">测试中碰到的其他问题</h2>
<p>可以看到脚本里已经设置了多次重试和超时重连,不过依然会收到写入超时和失败的返回,原来 Elasticsearch 默认对每个 node 做 segment merge 的时候,有磁盘保护措施,速度上限限制在 20MB/s。这在压测的时候就容易触发。</p>
<blockquote>
<p>[2015-01-10 09:41:51,273][INFO ][index.engine.internal ] [node1][logstash-2015.01.10][2] now throttling indexing: numMergesInFlight=6,maxNumMerges=5</p>
</blockquote>
<p>修改配置重启即可:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="s">indices.store.throttle.type:merge</span>
<span class="s">indices.store.throttle.max_bytes_per_sec:500mb</span>
</code></pre>
</div>
<p>关于这个问题,ES 也有讨论:<a href="https://github.com/elasticsearch/elasticsearch/issues/6081">Should we lower the default merge IO throttle rate?</a>。或许未来会有更灵活的策略。</p>
<p>更多 ES 性能测试和优化建议,参考:<a href="http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/indexing-performance.html">http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/indexing-performance.html</a></p>
LogStash::Outputs::ElaticSearch 使用 http 协议时的内存泄露问题
2015-02-10T00:00:00+08:00
logstash
ruby
elasticsearch
http://chenlinux.com/2015/02/10/logstash-outputs-elasticsearch-http-memory-leak
<p>Logstash 早年有三种不同的插件写数据到 Elasticsearch 中,分别采用 node,http 和 river 方式。从 1.4 版本以后,在重构的 LogStash::Outputs::ElasticSearch 插件中,通过 <code class="highlighter-rouge">protocol</code> 参数,完成了对多种方式的整合。其中,node 和 transport 方式,都是调用 Java 库的 API,而 http 方式,则调用的 REST API。</p>
<p>在 Elasticsearch 集群和 Logstash 集群不在一个网段的时候,一般都只能采用 REST API 写数据。而且根据测试情况,采用 http 方式的写入性能,也要稍微高过 node 方式,所以,我一直都推荐采用这种方式。不过随着系统的长期运行,却发现日志流转总是不太顺畅,实际写入 Elasticsearch 的数据慢慢的就会越来越少。因为 Logstash 本身内部并无缓存机制,所以比较难判断到底是哪步出了问题——甚至可能就是 Elasticsearch 在高负载情况下就写不动?</p>
<p>和 childe 聊了一下携程采用 transport 方式运行的情况,发现他们的 Elasticsearch 集群没有出现过类似越写越少的情况。把 logstash 的配置改成写文件,也一直没有再出现堵塞消息队列的情况。问题就此锁定在 logstash 写数据的 http 过程中。</p>
<p>进到源码目录里阅读<a href="https://github.com/elasticsearch/logstash/blob/1.4/lib/logstash/outputs/elasticsearch/protocol.rb#L67">相关代码</a>,发现在 <code class="highlighter-rouge">build_client</code> 方法里有很有趣的一段注释:</p>
<blockquote>
<h1 id="use-ftw-to-do-indexing-requests-for-now-until-we">Use FTW to do indexing requests, for now, until we</h1>
<p># can identify and resolve performance problems of elasticsearch-ruby</p>
</blockquote>
<p>这个好玩了。因为我在两年前用过官方出的 elasticsearch 的 Perl 客户端库,性能是非常不错的。怎么 Ruby 库会这么被嫌弃?</p>
<p>于是又切换到当前最新的 1.5.0beta1 版本看看这块是<a href="https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/protocol.rb#L70">怎么处理的</a>。最新版已经放弃了作者自己的 FTW 库,用上了官方的 Ruby 库,具体传输层用的是 JRuby 专有的 Manticore 库。</p>
<p>然后又发现 github 上几个相关的 issue:</p>
<ul>
<li><a href="https://github.com/elasticsearch/logstash/issues/1604">File descriptors are leaked when using HTTP</a></li>
<li><a href="https://github.com/elasticsearch/logstash/pull/1777">Add HTTP Auth and SSL to the ES output plugin</a></li>
<li><a href="https://github.com/cheald/manticore/wiki/Performance">manticore wiki/Performance</a></li>
</ul>
<p>所以问题很明确了,logstash-1.4.2 依赖的 ftw-0.0.39,有内存泄露问题。logstash 开发者在去年十一月升级了 ftw-0.0.40 解决这个问题,但是 logstash-1.4 那时候已经没有 release 计划了…… 差不多同时间,LogStash::Outputs::ElasticSearch 更换了底层 HTTP 依赖库为性能跟 FTW 相近的 Manticore,并且在前些天随 1.5.0beta1 版本发布。</p>
<p>升级成 1.5.0beta1 后,测试运行几天,Elasticsearch 的写入数据量一直没有下降。可以认定问题解决。</p>
<p>Logstash-1.5 和 Logstash-1.4 在 plugin API 方面没有什么变化,有写自己 plugin 的童鞋不用太过担心,可以放心测试然后升级使用。我目前发现的唯一一个变化就是:Logstash-1.5 改用 jackson 库替代原生 json 库了。所以原先可以直接:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">parsed</span> <span class="o">=</span> <span class="no">JSON</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
</code></pre>
</div>
<p>现在应该通过 logstash 内部方式调用:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">require</span> <span class="s1">'logstash/json'</span>
<span class="n">parsed</span> <span class="o">=</span> <span class="no">LogStash</span><span class="o">::</span><span class="no">Json</span><span class="p">.</span><span class="nf">load</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
</code></pre>
</div>
JRuby 调用 maxmind-java 测试
2015-01-22T00:00:00+08:00
logstash
geoip
ruby
performance
java
http://chenlinux.com/2015/01/22/logstash-maxmind-java
<p>GeoIP 是一个非常有用的信息,也是使用 ELKstack 时一般都会加上的过滤器插件。不过 geoip 插件的性能,有些时候却会成为整个系统的瓶颈。另一个问题,则是 GeoIP 数据文件的准确度,在国内比较头疼。即使你有一个自己处理出来的准确度较高的 IP 库,GeoIP 也没有提供现成的修改数据文件内容的工具。这个时候,MaxMind 公司的 GeoIP2 就进入我的视线了。</p>
<p>GeoIP2 在字段上比 GeoIP 更丰富。而且还提供了 <a href="https://metacpan.org/pod/MaxMind::DB::Writer">MaxMind::DB::Writer</a> 库方便使用者自己生成 GeoIP2 数据文件!感谢<a href="http://weibo.com/345198426">@纯白色燃烧</a>童鞋用<a href="http://blog.yikuyiku.com/?p=4144">自己的 CPAN 库成功倒逼</a> MaxMind 公司。</p>
<p>据@纯白色燃烧 介绍,GeoIP2 比 GeoIP 有六到七倍的性能提升。不过他是在 C 平台下,使用 libmaxminddb 库做的测试,而 logstash 是 JRuby 平台,所以我们需要的是验证如何在 JRuby 上使用 GeoIP2,以及跟 GeoIP 的性能对比。</p>
<p>在 JRuby 上用模块,有两种方式,一种是纯 Ruby 实现,一种是纯 Java 实现。MaxMind 提供了纯 Java 实现,社区另外有一个纯 Ruby 实现的库。下面开始测试。</p>
<h2 id="section">准备工作</h2>
<p>首先需要准备环境。安装 JRuby,纯 Ruby 实现的 maxminddb 库;然后下载 GeoIP2 数据文件,下载 Java 实现的 MaxMind-Java 库。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sudo port install jruby
sudo jgem install maxminddb
wget https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
gzip -d GeoLite2-City.mmdb.gz
wget https://github.com/maxmind/GeoIP2-java/releases/download/v2.1.0/geoip2-2.1.0-with-dependencies.zip
unzip geoip2-2.1.0-with-dependencies.zip
</code></pre>
</div>
<h2 id="section-1">测试程序</h2>
<p>准备就绪,然后就是如何测试的问题了。为了贴近 logstash 运行环境,我扒拉了一下 logstash 最核心的 <code class="highlighter-rouge">pipeline.rb</code> 文件,简化出来了一个测试程序。相当于是 <code class="highlighter-rouge">logstash -w 20 -e 'input {generator {}} filter {geoip{}} output {null{}}</code> 的效果:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/env jruby</span>
<span class="nb">require</span> <span class="s2">"geoip"</span>
<span class="nb">require</span> <span class="s2">"maxminddb"</span>
<span class="nb">require</span> <span class="s2">"thread"</span>
<span class="nb">require</span> <span class="s2">"java"</span>
<span class="c1"># 测试数据</span>
<span class="n">ip</span> <span class="o">=</span> <span class="s1">'202.106.0.20'</span>
<span class="c1"># 加载 maxmind-java 的所有 jar 包</span>
<span class="no">Dir</span><span class="p">[</span><span class="s2">"/Users/raochenlin/geoip2-2.1.0/lib/*.jar"</span><span class="p">].</span><span class="nf">each</span> <span class="p">{</span> <span class="o">|</span><span class="n">jar</span><span class="o">|</span> <span class="nb">require</span> <span class="n">jar</span> <span class="p">}</span>
<span class="c1"># 导入关键性的 java 类</span>
<span class="n">import</span> <span class="n">com</span><span class="p">.</span><span class="nf">maxmind</span><span class="p">.</span><span class="nf">geoip2</span><span class="o">.</span><span class="no">DatabaseReader</span>
<span class="n">import</span> <span class="n">java</span><span class="p">.</span><span class="nf">net</span><span class="o">.</span><span class="no">InetAddress</span>
<span class="c1"># 这个原生的 java 写法是:</span>
<span class="c1"># File database = new File("/Users/raochenlin/GeoLite2-City.mmdb")</span>
<span class="c1"># DatabaseReader reader = new DatabaseReader.Builder(database).build()</span>
<span class="c1"># 之前对 java 不太懂,想直接 import Builder 进来</span>
<span class="c1"># 其实 Builder 是DatabaseReader 类里的静态类(public final static class),不能直接 import</span>
<span class="n">database</span> <span class="o">=</span> <span class="n">java</span><span class="p">.</span><span class="nf">io</span><span class="o">.</span><span class="no">File</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="s2">"/Users/raochenlin/GeoLite2-City.mmdb"</span><span class="p">)</span>
<span class="vi">@reader</span> <span class="o">=</span> <span class="no">DatabaseReader</span><span class="o">::</span><span class="no">Builder</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">database</span><span class="p">).</span><span class="nf">build</span><span class="p">()</span>
<span class="c1"># 纯 Ruby 实现的库</span>
<span class="vi">@db</span> <span class="o">=</span> <span class="no">MaxMindDB</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="s1">'/Users/raochenlin/GeoLite2-City.mmdb'</span><span class="p">)</span>
<span class="c1"># 老的 GeoIP 库,需要制定不同的数据文件类型,这部分直接抄自 logstash 源码</span>
<span class="vi">@geo</span> <span class="o">=</span> <span class="no">GeoIP</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="s1">'/Users/raochenlin/Downloads/logstash-1.4.2/vendor/geoip/GeoLiteCity.dat'</span><span class="p">)</span>
<span class="vi">@geoip_type</span> <span class="o">=</span> <span class="k">case</span> <span class="vi">@geo</span><span class="p">.</span><span class="nf">database_type</span>
<span class="k">when</span> <span class="no">GeoIP</span><span class="o">::</span><span class="no">GEOIP_CITY_EDITION_REV0</span><span class="p">,</span> <span class="no">GeoIP</span><span class="o">::</span><span class="no">GEOIP_CITY_EDITION_REV1</span>
<span class="ss">:city</span>
<span class="k">when</span> <span class="no">GeoIP</span><span class="o">::</span><span class="no">GEOIP_COUNTRY_EDITION</span>
<span class="ss">:country</span>
<span class="k">when</span> <span class="no">GeoIP</span><span class="o">::</span><span class="no">GEOIP_ASNUM_EDITION</span>
<span class="ss">:asn</span>
<span class="k">when</span> <span class="no">GeoIP</span><span class="o">::</span><span class="no">GEOIP_ISP_EDITION</span><span class="p">,</span> <span class="no">GeoIP</span><span class="o">::</span><span class="no">GEOIP_ORG_EDITION</span>
<span class="ss">:isp</span>
<span class="k">else</span>
<span class="k">raise</span> <span class="no">RuntimeException</span><span class="p">.</span><span class="nf">new</span> <span class="s2">"This GeoIP database is not currently supported"</span>
<span class="k">end</span>
<span class="c1"># 开始 logstash 流程</span>
<span class="c1"># 创建从 input 到 filter 的缓冲队列,固定大小 20</span>
<span class="c1"># SizedQueue 是 thread 库导入的</span>
<span class="vi">@input_to_filter</span> <span class="o">=</span> <span class="no">SizedQueue</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span>
<span class="c1"># 具体的 geoip 过滤器线程</span>
<span class="k">def</span> <span class="nf">geoworker</span>
<span class="k">begin</span>
<span class="k">while</span> <span class="kp">true</span>
<span class="n">ip</span> <span class="o">=</span> <span class="vi">@input_to_filter</span><span class="p">.</span><span class="nf">pop</span>
<span class="c1"># GeoIP 查询方法</span>
<span class="c1"># data = @geo.send(@geoip_type, ip)</span>
<span class="c1"># puts data.to_hash[:city_name]</span>
<span class="c1"># MaxMind-java 查询方法,注意传入的是 InetAddress 对象</span>
<span class="n">data</span> <span class="o">=</span> <span class="vi">@reader</span><span class="p">.</span><span class="nf">city</span><span class="p">(</span><span class="no">InetAddress</span><span class="p">.</span><span class="nf">getByName</span><span class="p">(</span><span class="n">ip</span><span class="p">))</span>
<span class="c1"># puts data.getCountry().getName()</span>
<span class="c1"># maxminddb 查询方法</span>
<span class="c1"># data = @db.lookup(ip)</span>
<span class="c1"># puts data.country.name</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># 定义 input 线程,传入一百万次 IP 到缓冲队列</span>
<span class="n">lines_num</span> <span class="o">=</span> <span class="mi">1000000</span>
<span class="n">input</span> <span class="o">=</span> <span class="no">Thread</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="n">lines_num</span><span class="p">.</span><span class="nf">times</span><span class="p">.</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">i</span><span class="o">|</span>
<span class="vi">@input_to_filter</span><span class="p">.</span><span class="nf">push</span><span class="p">(</span><span class="n">ip</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># IP 发送完毕,计算每秒处理的速率</span>
<span class="n">end_time</span> <span class="o">=</span> <span class="no">Time</span><span class="p">.</span><span class="nf">now</span><span class="p">.</span><span class="nf">to_f</span> <span class="o">*</span> <span class="mi">1000</span>
<span class="nb">puts</span> <span class="n">lines_num</span> <span class="o">*</span> <span class="mi">1000</span> <span class="o">/</span> <span class="p">(</span><span class="n">end_time</span> <span class="o">-</span> <span class="vi">@start_time</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># 定义 filter 线程,启动 20 个</span>
<span class="n">arr</span> <span class="o">=</span> <span class="mi">20</span><span class="p">.</span><span class="nf">times</span><span class="p">.</span><span class="nf">collect</span> <span class="k">do</span>
<span class="no">Thread</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span>
<span class="n">geoworker</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># 记录开始时间,运行定义好的各线程</span>
<span class="vi">@start_time</span> <span class="o">=</span> <span class="no">Time</span><span class="p">.</span><span class="nf">now</span><span class="p">.</span><span class="nf">to_f</span> <span class="o">*</span> <span class="mi">1000</span>
<span class="n">input</span><span class="p">.</span><span class="nf">join</span>
<span class="n">arr</span><span class="p">.</span><span class="nf">each</span><span class="p">{</span><span class="o">|</span><span class="n">t</span><span class="o">|</span> <span class="n">t</span><span class="p">.</span><span class="nf">join</span><span class="p">}</span>
</code></pre>
</div>
<h2 id="section-2">测试结果</h2>
<p>在一百万次查询的测试中,结果如下:</p>
<ol>
<li>geoip worker 的查询 qps 是:6038.902610617599</li>
<li>maxminddb worker 的查询 qps 是:4621.093443130513</li>
<li>maxmind-java worker 的查询 qps 是:27943.88867154753</li>
</ol>
<p>可见,对于这部分有性能要求的,完全可以改用 <code class="highlighter-rouge">maxmind-java</code> 库,可以数倍提高。</p>
扩展 Zabbix Web 页面功能
2015-01-21T00:00:00+08:00
monitor
zabbix
php
http://chenlinux.com/2015/01/21/extends-zabbix-web
<p>zabbix 是目前非常流行的一个开源监控系统。虽然核心代码是 C 的,却通过 PHP 的 web 端提供了非常方便的界面和 RPC 接口。可以看到很多讲如何通过 RPC 接口自动化 zabbix 操作的文章。不过,如果你想做的事情正好没有现成的接口或者界面,怎么办呢?这时候就感谢 zabbix 的后端是用的 MySQL 数据库了,这意味着我们可以很方便的扩展 Zabbix 页面和接口的功能。</p>
<p>打个比方:<strong>我们一般都会按照 hostgroup 给某个 item 做一个 summary 汇总,然后针对 summary 的值来做报警。但是收到报警的时候,怎么能快速的知道这个 group 里是哪些 host 情况相对更严重呢?</strong></p>
<p>zabbix 页面和接口,都没有提供这种信息查看方式。所以,我们需要自动动手,实现这个功能。</p>
<p><a href="http://weibo.com/spider4k">@南非蜘蛛</a> 的 <a href="https://github.com/spider4k/zatree">zatree 项目</a>,解决了跟这个类似的问题。它的着手点是:针对 hostgroup 查看 graph,通过 graph 完成肉眼查看对比和 item 值的排序。但是,单个 graph 上可能就需要加载很多 item 信息。在 hostgroup 较大,或者单 host 监控项较多的情况下,zatree 直接就因为获取过多信息得不到 MySQL 响应变得无法正常访问了。</p>
<p>我的思路是:</p>
<ol>
<li>获取 hostgroup 列表供选择;</li>
<li>根据选择的 hostgroup 获取 item 列表;</li>
<li>根据选择的 hostgroup 和 item 获取全部 host 的 lastvalue 并排序;</li>
<li>排序后的 host 应该提供 history 的 graph 查看链接;</li>
<li>尽可能借用 zabbix-web 的界面。</li>
</ol>
<p>这其中的第 1、3 步都是有现成的 API 的,直接用 hostgroup.get 和 item.get 即可。主要说说第 2、5 步。</p>
<h2 id="api">新增 API</h2>
<p>前面说了,API 扩展其实就是通过 MySQL 操作完成。这里通过已知 groupid 获取 item 列表,放到 MySQL 里其实就是一行 select 语句:<code class="highlighter-rouge">SELECT DISTINCT key_ FROM items WHERE hostid IN (SELECT hostid FROM hosts_groups WHERE groupid='1');</code>。</p>
<p>而要实现在界面上,最简单的方式,参考 <code class="highlighter-rouge">include/items.inc.php</code> 里的 <em>get_item_by_hostid</em> 方法,可以定义函数如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>function get_items_by_groupid($groupid) {
$items = array();
$sql = 'SELECT DISTINCT key_ FROM items WHERE hostid IN (' .
'SELECT hostid FROM hosts_groups WHERE groupid=' . zbx_dbstr($groupid) .
')';
$db_items = DBselect($sql);
while ($item = DBfetch($db_items)) {
array_push($items, $item['key_']);
}
return $items;
}
</code></pre>
</div>
<p>这就可用了。</p>
<p>不过这个函数你只能在 require 了 items.inc.php 的 PHP 页面里使用,不能暴露成 RPC 接口。</p>
<h3 id="rpc-">修改为 RPC 接口</h3>
<p>首先要简介一下 zabbix 的 RPC 接口是怎么传递的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>api_jsonrpc.php
|-> api/rpc/class.cjsonrpc.php
|-> api/rpc/class.czbxrpc.php
|-> include/classes/api/API.php
</code></pre>
</div>
<p>在 API.php 中,通过 <code class="highlighter-rouge">getObjectClassName</code> 方法,在本文件里的 <code class="highlighter-rouge">$classMap</code> 获取对象的类名。</p>
<p>所以,添加一个接口,分为几步:</p>
<ol>
<li>实现新的类;</li>
<li>在 API.php 的 <code class="highlighter-rouge">$classMap</code> 里添加对应键值对;</li>
<li>在 API.php 中添加返回对应类的方法(这步是为了能在其他代码里用 <code class="highlighter-rouge">API::Item()</code> 这样的调用方式)。</li>
</ol>
<p>好,第一步,创建 <code class="highlighter-rouge">api/classes/CItemByGroup.php</code> 文件,内容如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="k">class</span> <span class="nc">CItemByGroup</span> <span class="k">extends</span> <span class="nx">CZBXAPI</span> <span class="p">{</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">get</span><span class="p">(</span><span class="nv">$groupid</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$items</span> <span class="o">=</span> <span class="k">array</span><span class="p">();</span>
<span class="nv">$sql</span> <span class="o">=</span> <span class="s1">'SELECT DISTINCT key_ FROM items WHERE hostid IN ('</span> <span class="o">.</span>
<span class="s1">'SELECT hostid FROM hosts_groups WHERE groupid='</span> <span class="o">.</span> <span class="nx">zbx_dbstr</span><span class="p">(</span><span class="nv">$groupid</span><span class="p">)</span> <span class="o">.</span>
<span class="s1">')'</span><span class="p">;</span>
<span class="nv">$db_items</span> <span class="o">=</span> <span class="nx">DBselect</span><span class="p">(</span><span class="nv">$sql</span><span class="p">);</span>
<span class="k">while</span> <span class="p">(</span><span class="nv">$item</span> <span class="o">=</span> <span class="nx">DBfetch</span><span class="p">(</span><span class="nv">$db_items</span><span class="p">))</span> <span class="p">{</span>
<span class="nb">array_push</span><span class="p">(</span><span class="nv">$items</span><span class="p">,</span> <span class="nv">$item</span><span class="p">[</span><span class="s1">'key_'</span><span class="p">]);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nv">$items</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>第二步,添加 <code class="highlighter-rouge">$classMap</code> 键值对,内容如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> 'itembygroup' => 'CItemByGroup',
</code></pre>
</div>
<p>第三步,添加对应方法,内容如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> /**
* @return CItemByGroup
*/
public static function ItemByGroup() {
return self::getObject('itembygroup');
}
</code></pre>
</div>
<p>这样,之前页面中直接使用 <code class="highlighter-rouge">get_items_by_groupid($groupid)</code> 的代码,就可以改写成:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$items = API::ItemByGroup()->get($groupid);
</code></pre>
</div>
<p>而在其他程序里,则可以用过 <strong>itembygroup.get</strong> 这个 RPC 接口获取相同结果了。</p>
<h2 id="zabbix-web--helper-">Zabbix Web 的布局和各种 helper 函数</h2>
<p>zatree 项目中完全自己写了整个页面,所以像授权啊、返回其他页啊都比较麻烦。所以我们尽量了解一下 zabbix web 本身是怎么写的页面,把数据融合到整体风格里面去。</p>
<p>其实 zabbix web 页面布局非常简单。主要分为三部分:</p>
<ol>
<li>include/page_header.php</li>
<li>new CWidget</li>
<li>include/page_footer.php</li>
</ol>
<p>header 和 footer 是很顾名思义的。不过 <code class="highlighter-rouge">page_header.php</code> 里,通过 <code class="highlighter-rouge">include/menu.inc.php</code> 的 <code class="highlighter-rouge">zbx_construct_menu()</code> 方法,会校验访问者的权限。</p>
<h3 id="section">新增页面授权</h3>
<p><code class="highlighter-rouge">menu.inc.php</code> 也很简单,跟前面 api 类似,也是一个大变量来控制菜单和页面的权限,这个变量叫 <code class="highlighter-rouge">$ZBX_MENU</code>。<code class="highlighter-rouge">$ZBX_MENU</code> 数组存放的,就是 zabbix web 顶部菜单大家看到的那几个标签,Monitoring、Report 等等。如果打算把页面加在顶部菜单上,那么就直接添加一个元素到 <code class="highlighter-rouge">$ZBX_MENU</code> 数组,如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> 'sort' => array(
'label' => _('Sort'),
'user_type' => USER_TYPE_ZABBIX_USER,
'default_page_id' => 0,
'force_disable_all_nodes'=> true,
'pages' => array(
array(
'url' => 'sort.php','label' => _('Sort')
)
)
),
</code></pre>
</div>
<p>如果打算加到到次级菜单,比如放到 Monitoring 下面,那么找到 <code class="highlighter-rouge">view</code> 元素(其 label 为 “Monitoring”),在其 <code class="highlighter-rouge">pages</code> 数组里加上即可:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> 'pages' => array(
...
array(
'url' => 'sort.php',
'label' => _('Sort'),
)
)
),
</code></pre>
</div>
<h3 id="cwidget-">CWidget 及其他组件</h3>
<p>zabbix 虽然没有使用特别明确的 MVC 框架,倒也不用大家到处自己去拼接输出 HTML 代码,它已经实现了很多 helper 函数。</p>
<p>比如:</p>
<ul>
<li>group 和 item 的选择器,可以用 <code class="highlighter-rouge">CComboBox()</code> 生成;</li>
<li>页面交互的表单,可以用 <code class="highlighter-rouge">CForm()</code> 生成;</li>
<li>数据展示的表格,可以用 <code class="highlighter-rouge">CTableInfo()</code> 生成;</li>
<li>history graph 的链接,可以用 <code class="highlighter-rouge">CLink()</code> 生成;</li>
</ul>
<p>然后,<code class="highlighter-rouge">CTableInfo()</code> 可以 <code class="highlighter-rouge">->addRow()</code>;<code class="highlighter-rouge">CForm()</code>、 <code class="highlighter-rouge">CComboBox()</code> 和 <code class="highlighter-rouge">CWidget()</code> 都可以 <code class="highlighter-rouge">->addItem()</code>。</p>
<p>把各种元素都添加到 CWidget 里以后,调用 <code class="highlighter-rouge">->show()</code> 方法即可。</p>
<p>此外,还提供有 <code class="highlighter-rouge">check_fields</code>, <code class="highlighter-rouge">get_request</code>, <code class="highlighter-rouge">validate_sort_and_sortorder</code>, <code class="highlighter-rouge">getPageSortOrder</code>, <code class="highlighter-rouge">make_sorting_header</code> 和 <code class="highlighter-rouge">order_result</code> 等方法帮助处理请求参数和数据表格展示。</p>
<p>最后效果如下:</p>
<p><img src="/images/uploads/zabbix_sort_web.jpg" alt="" /></p>
给 Kibana3 添加脚本化字段支持
2015-01-06T00:00:00+08:00
logstash
kibana
elasticsearcch
javascript
http://chenlinux.com/2015/01/06/implement-script-field-for-kibana3
<p>Kibana4 中确实有不少让人眼前一亮的新特性,但是整体框架和使用思路上的重构实在让人较难上手。所以,把一些有需要的特性,port 回目前更稳定的 Kibana3 就有必要了。好在去年在自己 fork 中已经做了很多铺垫,包括一些基础库的版本更新。这些特性基本都只需要几行代码的变动就可以实现。</p>
<p>从上次写博客介绍的 uniq histogram 去重统计功能后,这段时间又添加了两个功能。</p>
<h2 id="table-">table 的数据导出</h2>
<p>kibana3 已经带有 <a href="https://github.com/eligrey/FileSaver.js">filesaver.js</a>,所以加一个 <code class="highlighter-rouge">exportAsCsv</code> 函数即可。要点在于怎么给 table panel 右上角那排小按钮加上一个新图标。</p>
<p>我之前说过,kibana3 代码划分的很细致,每个 panel 都固定只需要提供 editor.html,module.html,module.js 三个文件即可。panel 本身的框架,是不用关心的。因为这部分代码,在 <code class="highlighter-rouge">app/directives/kibanaPanel.js</code> 中。这次我们想修改 panel 外围的样式,就需要来看这个的代码了。最关键的部分在这里:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="s1">'<span ng-repeat="task in panelMeta.modals" class="row-button extra" ng-show="task.show">'</span> <span class="o">+</span>
<span class="s1">'<span bs-modal="task.partial" class="pointer"><i '</span> <span class="o">+</span>
<span class="s1">'bs-tooltip="task.description" ng-class="task.icon" class="pointer"></i></span>'</span><span class="o">+</span>
<span class="s1">'</span>'</span> <span class="o">+</span>
</code></pre>
</div>
<p>也就是说,它会读取你在 module.js 里定义的 <code class="highlighter-rouge">$scope.panelMeta.modals</code> 数组,然后依次显示。那么就好办了,在我们 table/module.js 里定义下就好了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nx">$scope</span><span class="p">.</span><span class="nx">panelMeta</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">modals</span> <span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="na">description</span><span class="p">:</span> <span class="s2">"Export"</span><span class="p">,</span>
<span class="na">icon</span><span class="p">:</span> <span class="s2">"icon-download-alt"</span><span class="p">,</span>
<span class="na">partial</span><span class="p">:</span> <span class="s2">"app/panels/table/export.html"</span><span class="p">,</span>
<span class="na">show</span><span class="p">:</span> <span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">exportable</span>
<span class="p">},</span>
</code></pre>
</div>
<p>为了跟其他的比如 inspector, editor 图标行为一致,这里又新增了一个 <code class="highlighter-rouge">$scope.panel.exportable</code> 变量。而这也带来一个问题:之前已经存在的 dashboard,他们的 schema 里是没有这个变量的,所以即便使用带有这个特性的 kibana 打开老 dashboard,依然看不到导出按钮。这时候,可以手动修改一下 schema 的 JSON 内容,添加上一行 <a href="https://github.com/chenryn/kibana-authorization/blob/master/src/app/dashboards/logstash.json#L138"><code class="highlighter-rouge">"exportable": true</code></a>,也可以点击 panel 上的 dup 复制按钮,复制出来的 panel 会读取默认变量设置,就会出现导出按钮了。然后删掉原 panel ,保存 dashboard 即可。</p>
<p><strong>注意</strong>:导出的数据只是 table 里的内容,这只是一个 js 功能。不要把它理解成调用 scroll API 获取 Elasticsearch 集群里的全部数据。</p>
<h2 id="scriptfield-">scriptField 聚合</h2>
<p>Kibana4beta3 的另一个重要特性,是可以预定义一段 script 为 scriptedField,然后在搜索、聚合的时候可以当做普通 field 一样使用这个 scriptedField。示例见官方博客说明(可以直接看<a href="http://chenlinux.com/2014/12/19/kibana-4-beta-3-now-more-filtery/">我的翻译</a>)。至于 script 本身能在 Elasticsearch 里做些什么,之前博客里也写过<a href="http://chenlinux.com/2014/11/27/elasticsearch-scripts-aggregations/">两个小示例</a>。</p>
<p><em>动态 script 功能在 ES 1.4 之前是因为安全问题被建议关闭的。1.4 开始加入了沙箱功能,才这么大胆的使用。</em></p>
<p>我印象中 script field 应该是不能保存在 mapping 里的,于是稍微看了一下 kibana4 的代码,疑似是另外用一个索引来存储这个信息。<em>不确保是这样,kibana4 的代码比 kibana3 难懂多了。</em></p>
<p>kibana3 整个界面结构跟 kibana4 不一样,没有单独的字段管理页面,而是通过 <code class="highlighter-rouge">app/services/fields.js</code> 提供了 <code class="highlighter-rouge">fields.list</code> 在各个 panel 的 editor.html 里做 <code class="highlighter-rouge">bs-typeahead</code>。所以,如果完整的思路 port 回来,应该是写一个 <em>app/services/scriptFields.js</em> 来提供 scriptedField 的增删改查,然后还要自己写个页面来提供操作界面。</p>
<p>作为页面手残党,我迅速决定放弃这个思路,选择一个更简单的方式来完成类似目的:直接在最常用的 terms panel 里提供输入 script 字符串的功能,反正每个 dashboard 最后会固化成 JSON 的。而且其他 panel 应该不太会用到这个功能(如果要在 table 里也实现,改动又稍大了。Kibana4 里我猜测应该是直接返回勾选的 fields,这个接口是支持 script 的;Kibana3 里则是返回全部字段,然后在 js 里完成的表格字段选择性展示)。</p>
<p>terms panel 中对类似情况就有示例在。这里本是有个 <code class="highlighter-rouge">tmode</code> 参数,用来选择是用 termsFacet 还是 termstatsFacet API。照葫芦画瓢,我新加了一个 <code class="highlighter-rouge">fmode</code> 参数,用来选择是普通字段(“normal”)还是脚本字段(“script”):</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nt"><div</span> <span class="na">class=</span><span class="s">"editor-option"</span> <span class="na">ng-show=</span><span class="s">"panel.fmode == 'script'"</span><span class="nt">></span>
<span class="nt"><label</span> <span class="na">class=</span><span class="s">"small"</span><span class="nt">></span>ScriptField<span class="nt"></label></span>
<span class="nt"><input</span> <span class="na">type=</span><span class="s">"text"</span> <span class="na">class=</span><span class="s">"input-large"</span> <span class="na">ng-model=</span><span class="s">"panel.script"</span> <span class="na">ng-change=</span><span class="s">"set_refresh(true)"</span><span class="nt">></span>
<span class="nt"></div></span>
</code></pre>
</div>
<p>然后在生成 request 的时候,做一下判断:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">if</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">fmode</span> <span class="o">===</span> <span class="s1">'script'</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">terms_facet</span><span class="p">.</span><span class="nx">scriptField</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">script</span><span class="p">)</span>
<span class="p">}</span>
</code></pre>
</div>
<p>这就 OK 了~</p>
<p>接下来另一个难点:<strong>terms panel 是支持点击生成 filtering 过滤条件的。</strong></p>
<p>显然 filtering 里没有 script 的支持。filtering 的功能都出自 <code class="highlighter-rouge">app/services/filterSrv.js</code> 服务。其中 <code class="highlighter-rouge">toEjsObj</code> 方法调用不同的 Elastic.js 的 Filter 方法。在这里面可以看到原本 terms 的是怎么生成的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">case</span> <span class="s1">'terms'</span><span class="err">:</span>
<span class="k">return</span> <span class="nx">ejs</span><span class="p">.</span><span class="nx">TermsFilter</span><span class="p">(</span><span class="nx">filter</span><span class="p">.</span><span class="nx">field</span><span class="p">,</span><span class="nx">filter</span><span class="p">.</span><span class="nx">value</span><span class="p">);</span>
</code></pre>
</div>
<p>那么我就添加一个:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">case</span> <span class="s1">'script'</span><span class="err">:</span>
<span class="k">return</span> <span class="nx">ejs</span><span class="p">.</span><span class="nx">ScriptFilter</span><span class="p">(</span><span class="nx">filter</span><span class="p">.</span><span class="nx">script</span><span class="p">);</span>
</code></pre>
</div>
<p>filterSrv 支持搞定。最后一步,就是返回 terms panel 的 module.js 里完成调用。过一遍 click 关键字很容易找到 <code class="highlighter-rouge">build_search</code> 方法。其中原先是这么生成过滤的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">if</span><span class="p">(</span><span class="nx">_</span><span class="p">.</span><span class="nx">isUndefined</span><span class="p">(</span><span class="nx">term</span><span class="p">.</span><span class="nx">meta</span><span class="p">))</span> <span class="p">{</span>
<span class="nx">filterSrv</span><span class="p">.</span><span class="nx">set</span><span class="p">({</span><span class="na">type</span><span class="p">:</span><span class="s1">'terms'</span><span class="p">,</span><span class="na">field</span><span class="p">:</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">field</span><span class="p">,</span><span class="na">value</span><span class="p">:</span><span class="nx">term</span><span class="p">.</span><span class="nx">label</span><span class="p">,</span>
<span class="na">mandate</span><span class="p">:(</span><span class="nx">negate</span> <span class="p">?</span> <span class="s1">'mustNot'</span><span class="p">:</span><span class="s1">'must'</span><span class="p">)});</span>
</code></pre>
</div>
<p>那么在这个前面判断一下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">if</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">fmode</span> <span class="o">===</span> <span class="s1">'script'</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">filterSrv</span><span class="p">.</span><span class="nx">set</span><span class="p">({</span><span class="na">type</span><span class="p">:</span><span class="s1">'script'</span><span class="p">,</span><span class="na">script</span><span class="p">:</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">script</span> <span class="o">+</span> <span class="s1">' == \"'</span> <span class="o">+</span> <span class="nx">term</span><span class="p">.</span><span class="nx">label</span> <span class="o">+</span> <span class="s1">'\"'</span><span class="p">,</span>
<span class="na">mandate</span><span class="p">:(</span><span class="nx">negate</span> <span class="p">?</span> <span class="s1">'mustNot'</span><span class="p">:</span><span class="s1">'must'</span><span class="p">)});</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span><span class="p">(</span><span class="nx">_</span><span class="p">.</span><span class="nx">isUndefined</span><span class="p">(</span><span class="nx">term</span><span class="p">.</span><span class="nx">meta</span><span class="p">))</span> <span class="p">{</span>
</code></pre>
</div>
<p>大功告成!</p>
<p><img src="http://ww4.sinaimg.cn/large/3dbd9afagw1eo07bw1ygsj20eh0bxjs6.jpg" alt="" /></p>
2014 年度个人总结
2014-12-26T00:00:00+08:00
http://chenlinux.com/2014/12/26/report-of-this-year
<p>又到一年底,总结个人业绩和得失的时候了。于我个人而言,2014 年真是精彩纷呈。</p>
<h2 id="section">说说写作</h2>
<p>首先,4 月份我写的《网站运维技术与实践》面市。开卖之前,有好心的朋友叮嘱说,万一碰到在网店评论区捣乱的,千万不要理会。不过半年多过去,似乎最差的评论也只是说章节名字取得太烂。这点真心承认,尤其是第一章,各小节标题直接就是各种 linux 命令,偏巧一般网上目录介绍默认就只显示最前面一点而不是展开全部目录的,第一眼看过去就好像本书是一本命令大全!</p>
<p>第一次写书,留下几个遗憾。第一,忘了写致谢!第二,章节的安排次序在我心中其实是有一个完整的先后逻辑的,然而我竟然忘记在前言中讲明。直到半年后有人问及为什么没有说说一个产品交付运维的完整流程应该如何,我回答说:看书目录的次序就是了。</p>
<p>好在销量据说尚可,这应该让我有机会在第二版中弥补。</p>
<p>为此书写的《西江月》最后是印在了封底内页,或许第二版,我给每章写一句诗做副标题?</p>
<p>其次,跟刘宇、长元、春生一起翻译的《Puppet Cookbook》应该也快面市了。一本 200 页的小书,编辑也忍耐我们这么多人来打酱油,呼呼~ Puppet 本身是一个到处都有新矿可挖的生态圈。曾经有同事看过刘宇的《Puppet 实战》和我的《网站运维技术与实践》Puppet 章节后说:你写的 puppet 跟刘宇的完全不一样的技术点,完全可以叫《Another Puppet 实战》。那么,现在大家有福了,这本翻译的 cookbook 内容大多又是我们两个之前没覆盖到的,可算是《Yet Another Puppet 实战》。</p>
<p>在这两个之外,还在 gitbook.com 上写了两本电子书,是关于实时大数据处理 ELKstack 的。稍后再单独说。</p>
<p>技术写作这件事情,本身在圈内就很有争议。比如左耳朵耗子就多次明嘲暗贬。我还是觉得,抛开赚钱的话题(其实就是根本不赚钱,一本书稿费还不抵作者半个月薪水呢),认认真真给自己的技术归纳体系,列目录,写总结,完善用例,深挖根源,一般情况下绝对是很难真的动手而且万难坚持下来的事情。一旦你决定要写成“书”而不是散落的“文本”,各方面的压力就从此变成动力。</p>
<p>突然想起来这段感慨在去年应该就写过了……因为《网站运维技术与实践》其实在去年秋天就完稿的。</p>
<h2 id="section-1">读书</h2>
<p>之前听过一句话:读书学习是性价比最高的个人投资。如果说之前我还习惯看 pdf 的话,随着自己拿到第一笔也是目前为止唯一一笔稿费,深刻感受到作者们辛苦劳动的不值钱,现在已经很主动的买书了。今年一共买了《Go 语言程序设计》《预测背后的逻辑》《机器学习实践指南》《金融时间序列分析》《时间序列预测实践教程》《Python 自动化运维》《日志管理与分析权威指南》《链接》《失控》《Time Management for System Administrators》《黑天鹅》《反脆弱》十二本纸质书,《神经漫游者》《追风筝的人》《汉语词律学》《汉语韵律词研究》四本电子书,借阅了《Splunk 大数据分析》《Docker 技术入门与实践》《Go 语言编程》三本书,受赠《深度解析SDN: 利益、战略、技术、实践》《设计之下: 搜狐新闻客户端的用户体验设计》《MacTalk 人生元编程》《互联网创业密码》《Zabbix 监控系统深度实践》《进化: 我们在互联网上奋斗的故事》《单页 Web 应用: JavaScript 从前端到后端》《Elasticsearch Server》《Puppet Cookbook》九本书。</p>
<p>合计二十八本,看完十二本,占 42.8%。明年希望尽力提高买书的有效利用率~</p>
<h2 id="section-2">工作和生活</h2>
<p>5 月去了苏州旅游。作为一个婚假都忘了休的人,这真是一次美好的记忆。原先计划要跑遍长三角,结果看着苏州园林,听着吴侬软语,吃着生煎,就挪不动步子,彻彻底底在苏州呆完了整个假期。现在一边写着这份总结,一边又想起昆曲博物馆里坐我左手边的那个台湾老教授,想起虎丘山门外林立的名家碑文。这是怎样一种奢侈。</p>
<p>7 月换了份工作到新浪。新浪的面试官是惯来喜欢砸人的,于是我先前在一家公司呆久了,就会先找个新浪的面试,被砸一砸,然后回去就可以安心的继续工作或者研究。不料这次真的进新浪了,开始了我“砸”别人的日子……</p>
<p>9 月是最紧张的时候,丢掉之前各种规划,接手一个没有测试报告,没有设计文档,没有运行状态,只有“又不行了”的日志系统。老妈正好这个月来北京,于是一边想着不能让老妈觉得我其实一直这么苦逼的拼命啊,一边半夜三点回家……</p>
<p>这破烂状态虽然现在结束了,但是手头已有和要有的这摊子事情,依然都是没测试没设计没文档的状态,真心要吐槽,一点“大公司”的感觉都没有啊。</p>
<h2 id="section-3">社区活动</h2>
<p>4 月 CSDN 邀请了 Larry Wall 来中国。对于 Perl 程序员简直是再幸福不过了。好玩的是教主在我的大骆驼书上签名时划破了那页纸,于是他拿起他的大骆驼印章,给那页上一口气按了十多个骆驼==!</p>
<p>12 月,主动提起应该继续 PerlChina 的 Advent 活动,在 fayland 的帮助下,搭建了 <a href="http://advent.perl-china.com">http://advent.perl-china.com</a> 网站,而且 24 篇 gift 我写了 11 篇。还是那句话,坚持是最大的困难……</p>
<p>同样在坚持的,还有 @perldaily 这个微博号,一年来每周的 perlweekly、rubyweekly、devopsweekly,都坚持阅读,并且挑选转发到微博上。12 月更是同时阅读着每天的 perladvent、perl6advent、catalystadvent、danceradvent、perladvent.kr、perladvent.jp、rubyadvent、goadvent、sysadvent、performanceadvent,并且转发到微博。</p>
<h2 id="section-4">技术动态</h2>
<h3 id="docker">docker</h3>
<p>年初的时候跟着去年下半年的惯性,还是很积极的跟踪尝试 docker 来着。包括用 docker 做了一个类似 JSFiddle 的 Perl 在线代码调试工具。唯一的问题就是 fork 炸弹,然后靠 ulimit 启动解决。</p>
<p>稍后还参加了第一次 docker beijing meetup。差不多时间接赵鹏的邀请试用了一把他的 visualops,转身自己用开源的 diagramo 试了试如何在页面拖动服务器图标生成 fig.yml 配置。不过玩起来好搞,搞成产品,那就难了,visualops 做的是真到位,赞!</p>
<p>docker 的故事就到这里,之后就没机会再参与了。</p>
<h3 id="perl">perl</h3>
<p>模仿 serverspec 工具写了 Rex::Test::Spec 模块,结果被 rex 项目作者邀请加入了 RexOps 开发组。不过实话是 Perl 确实现在式微,2013 年,Rex 跟 Saltstack、Ansible 感觉都是差不多的小众产品,到今年,后二者风头正劲,无数人开始问“salt 跟 puppet 哪个好啊”的问题。公司内部也没有 Perl 氛围,我也就保持着自己个人使用,懒得推广了。</p>
<p>另一个一直在保持跟踪的是 Perl6。测试过用 Perl6 写 Puppet 的 ENC 脚本,还为此去修复了 Perl6 版本的 YAML::Dump 模块。测试过用 Perl6 如何做并发编程,了解了 Promise、Supply 等概念。但愿教主和 jnthn 能在明年解决一定的性能问题,发布 6.0 版吧……</p>
<p>今年还订阅了 Perl5Porter 的邮件组,看着 Perl5 开发者们是如何维护 Perl5 代码的。跟昨天 Larry Wall 发表在 Perl6 Advent 上的想法真是出奇的一致:Perl 是一个健全的城市,不需要五年计划,有人愿意盖房子,市议会负责别让他影响其他人就够了。就在这个思想的指导下,今年 5 月发的 Perl5 version 20 加上了 sub signature,实现者是今年 2 月份才提出自己要做的;而下半年突然出现的俄罗斯大神则提出要给 Perl5 的 OOP 性能提高一倍,然后看着 P5P 的人一步一步教他怎么用 git,怎么拆分他的大 patch 成一个一个 commit 和 test,让人无比期待明年的 Perl5 version 22 了。</p>
<h3 id="elkstack">ELKstack</h3>
<p>ELKstack 在今年占据了我大量的精力,从博客中就可以看到。2014 年,一共发了 64 篇博客,标记为 ELK 相关的有 27 篇,接近一半。</p>
<p>ES 公司从今年 4 月开始停止了 Kibana3 的开发,专门去做 Kibana4 的重构工作,至今还没发布正式版。在这大半年的空档期内,我在自己的 <a href="https://github.com/chenryn/kibana-authorization">fork 仓库</a>里,新增了 11 项功能,替换 Facet 为 Aggr 接口,百分比统计、区间分布统计、去重数据走势、高德地图、请求生成器、阈值通知、数值统计值地图、单图表引用、表单导出等等。还提供了社区最完整的验证授权代理功能。目前收到了 40 个 star。</p>
<p>去年底建的 QQ 群,到目前有接近 400 人加入。尤其开心的,这让我发现 ELK 的使用者,很多是开发工程师、安全工程师。这种交叉领域的聊天非常舒服,给人启发。当然要感谢携程的几位朋友,wood 童鞋老早在群里公开自己的十亿级用例的 ppt,也是官网文档的活字典,childe 童鞋最早开始写 statisictrend panel,没他吃螃蟹在先,我可能还想不到自己动手做 kibana 去。</p>
<p>QQ 群里经常出现的重复问题,也触发我最终选择在 gitbook.com 上写电子书。很遗憾 ELK 还不够火,所以单独写纸质书的可能性是微乎其微了,好在 gitbook 的使用感觉还不错,需要吐槽的就是定价只能涨不能降这个设定,此外,不凑够 $50 不能取现,不取现不能删除书籍也让我头疼不已,我真的不是有意给自己电子书设置价格的。</p>
<p>两本书的 markdown 源码都发在了 github 上托管。分别有 <a href="https://github.com/chenryn/logstash-best-practice-cn">82</a> 和 <a href="https://github.com/chenryn/kibana-guide-cn">34</a> 个 star。此外,还收到了共计 573.87 元支付宝打赏。然后我花了其中两百多去买了一个很有意思的域名:<a href="http://kibana.logstash.es/">kibana.logstash.es</a>。哈哈~</p>
<p>ELK 本身的讨论和思考,年终总结里就不再啰嗦了,基本都写在电子书里,欢迎大家阅读、点赞和打赏……</p>
<p>10 月,medcl 主办了 ES 中国的第三次大会,也是第一届正式的大会(突然有第 24 次第一届搞笑诺贝尔奖即视感)。应该有 200 人到会场。我做了 《<a href="http://pan.baidu.com/s/1i3qsoBF#path=%252FESCC%25233">{<span>{</span>More}} Kibana</a>》的分享。认识了几位演讲嘉宾,一个赛一个的年轻,全都是 85 后。</p>
<p>11 月,长元离京前的 Puppet 群组 8 人小聚会上,分享 ELK 概念和演示常见配置用法。</p>
<p>12 月,Beijing.pm 例行月度 7 人小聚会上,分享 ELK 概念和演示常见配置用法。</p>
<p>加上明年 1 月准备在<a href="http://www.uml.com.cn/communicate/2015-1-17.asp">火龙果</a>上做的 ELK 分享。这会是连续 4 个月在外分享 ELKstack 了。这或许又会是一种坚持?看看明年 2 月以后还有没有机会继续吧……</p>
【翻译】Kibana 4 beta 3 发布,重新支持过滤器
2014-12-19T00:00:00+08:00
logstash
kibana
http://chenlinux.com/2014/12/19/kibana-4-beta-3-now-more-filtery
<p>本文是 Elasticsearch 官方博客内容,原文地址:<a href="http://www.elasticsearch.org/blog/kibana-4-beta-3-now-more-filtery/">http://www.elasticsearch.org/blog/kibana-4-beta-3-now-more-filtery/</a></p>
<hr />
<p>Kibana 4 Beta 3 出来啦! 我们依然给你机会直接<a href="http://www.elasticsearch.org/overview/kibana/installation/">下载 Kibana 4 Beta 3</a>。不过还是要建议你阅读本文对主要特性的讲解。嗯,先暂停一下下载,开始阅读吧!</p>
<h2 id="section">交互式图表和仪表盘</h2>
<p>过滤器回到了仪表盘上,也可以在单个可视化页上使用了!柱状图、点图、饼图都可以通过点击的方式创建可切换的过滤器。我们还添加了一些函数来操作所有的过滤器,这样你可以一键切换整个过滤效果。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/12/Screen-Shot-2014-12-15-at-12.28.30-PM-1024x693.png" alt="Screen Shot 2014-12-15 at 12.28.30 PM" /></p>
<h2 id="section-1">脚本化字段</h2>
<p>Kibana 现在支持 Elasticsearch 脚本了!不单是可以写脚本,还可以给它命名,并且在应用中跟用普通字段一样调用你取的名字。创建一个脚本化字段,这个字段就像本来就存在一样的显示在你的 Kibana 文档里了。唯一需要注意的是,脚本毕竟不是 Elasticsearch 索引的内容,你不能在这个字段里进行搜索。</p>
<p>你可以用脚本来连接多个字段,或者在数值字段上做运算,然后把结果导入可视化页里。为了帮助你上手,我们在脚本化字段屏下添加了一个标题叫“从时间字段创建的示例”的连接。你可以在设置(Settings)标签页的索引(Index)区域里找到这个连接。选择或者创建一个索引表达式,然后点击“脚本化字段(Scripted Fields)”标签。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/12/Screen-Shot-2014-12-15-at-1.06.51-PM.png" alt="Screen Shot 2014-12-15 at 1.06.51 PM" /></p>
<p>做完这些以后,你就可以在聚合页里找到一些新的数值字段可用。比如说,我们可以查一天的 24 个小时,然后获取 30 天 来每个小时的 hits 数的总和:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/12/unnamed.png" alt="unnamed" /></p>
<h2 id="source-">高亮和 _source 的新格式</h2>
<p>JSON 很棒,我们都爱 JSON。谁会不爱 JSON 呢?XML,这是谁?完全无关紧要嘛。</p>
<p>JSON 在查看上可能有点乱,所以我们对格式做了一点优化。原始的 JSON 内容,当然可以在点击 JSON 标签展开事件后查看。Kibana 现在还会自动高亮匹配上的字段,甚至把他们挪到本行开头的位置展示:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/12/Screen-Shot-2014-12-16-at-11.16.17-AM-1024x730.png" alt="Screen Shot 2014-12-16 at 11.16.17 AM" /></p>
<h2 id="hit-">hit 连接</h2>
<p>可能你已经注意到前面截屏上的 “Link to..” ? 你可能不需要分享整个可视化结果或者一个搜索结果,你只是想让别人看到一条重要的命中的记录。现在,这事儿简单了!</p>
<h2 id="metric-visualization">metric visualization</h2>
<p>有时候你不需要图或者文档!你只需要一个数值在仪表盘上就够了。现在可以做到了:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/12/Screen-Shot-2014-12-16-at-11.16.59-AM.png" alt="Screen Shot 2014-12-16 at 11.16.59 AM" /></p>
<p>好了,就是这些!还是那句话,到 <a href="https://github.com/elasticsearch/kibana">GitHub</a> 上给我们提问题,建议,贡献。或者,如果你跟我们一样喜欢 IRC,加入我们在 Freenode 上的 #kibana 频道。</p>
Kibana 中几个不同的 filtering
2014-12-08T00:00:00+08:00
logstash
elasticsearch
kibana
http://chenlinux.com/2014/12/08/difference-filterings-kibana
<p>用过 kibana 的都知道,kibana 的图表上,可以直接点击某个值,就能自动添加这个过滤条件到 filtering 里,然后整个 dashboard 上所有的图表都会刷新成在这个过滤条件下的新状态。但是如果你要想自己手动添加 filtering 的时候,就会发现,自己添加的,写法好像跟自动生成的长得不太一样。</p>
<p>而今天,我在同事的提醒下,发现更进一步的情况,即使都是通过点击图表添加上的 filtering,其实长得也不一样,如下图:</p>
<p><img src="/images/uploads/filterings.png" alt="" /></p>
<ul>
<li>在 histogram 面板上拖拽鼠标,生成的是 range filtering</li>
<li>在 terms 面板上点击某个值,生成的是 term filtering</li>
<li>在 table 面板左侧列表上点击某个字段,浮出的小面板里点击某个值,生成的是 query filtering</li>
<li>在 filtering 手工添加,生成的是 query_string filtering</li>
</ul>
<p>这几个页面上的不同,反应在实际的请求 JSON 里又有什么区别呢?</p>
<p>我们可以点开面板右上角的 inspect 按钮看生成的 curl 命令。其中 filtering 部分如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="w"> </span><span class="s2">"filter"</span><span class="err">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"bool"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"must"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nt">"range"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"@timestamp"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"from"</span><span class="p">:</span><span class="w"> </span><span class="mi">1418009781101</span><span class="p">,</span><span class="w">
</span><span class="nt">"to"</span><span class="p">:</span><span class="w"> </span><span class="s2">"now"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nt">"terms"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_type"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"mweibo_webinf"</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nt">"fquery"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"query"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"query_string"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"query"</span><span class="p">:</span><span class="w"> </span><span class="s2">"host:(\"web093.mweibo.tc.sinanode.com\")"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"_cache"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nt">"fquery"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"query"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"query_string"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"query"</span><span class="p">:</span><span class="w"> </span><span class="s2">"host:\"web093.mweibo.tc.sinanode.com\""</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"_cache"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
</div>
<p>前面两个不出意外,都是很标准的 api 示例的样子。比较特殊的是后面两个:</p>
<p>第三个其实就是通过 table 左侧字段菜单点出来的,虽然通过鼠标点击操作,只可能生成一个单一的键值查询,但这里却给加上了一对小括号!这是完全没有必要的,简直可以怀疑是不是当初开发人员手抖了……</p>
<p>当然,并不是说这种生成完全没有用。比方说,其实你本来是打算查询来自两台机器的日志。如果没想到用括号,可能直接在 query_string 里就写 <code class="highlighter-rouge">host:"web001" OR host:"web002"</code> 了。但是在这个 query filtering 里,因为页面上已经有单独填字段的地方了。那就只用在 query 那栏写 <code class="highlighter-rouge">"web001" OR "web002"</code> 好了。</p>
<p>以上。不过我依然怀疑是开发人员手抖。</p>
利用脚本灵活定制 Elasticsearch 中的聚合效果
2014-11-27T00:00:00+08:00
logstash
elasticsearch
groovy
http://chenlinux.com/2014/11/27/elasticsearch-scripts-aggregations
<p>这几天阅读 Splunk 书,发现 Splunk 作为一个不需要提前结构化数据的处理工具,在自动发现的 “interesting fields” 以外,也提供了在页面通过正则临时产生新字段的能力。类似下面这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sourcetype="impl_splunk_gen"
| rex "ip=(?P<subnet>\d+\.\d+\.\d+)\.\d+"
| chart values(subnet) by user network
</code></pre>
</div>
<p>这就蛮让人流口水的了。毕竟谁也不可能保证自己在结构化的时候做到了万事俱备。不过,ELK 虽然建议大家在 logstash 里通过 grok 来预处理,其实本身也是有这个能力的。今天稍微测试了一下,通过 ES 的 <strong>scripting</strong> 模块,完全可以实现这个效果。</p>
<p><em>测试在 Elasticsearch 1.4.1 上进行。较低的版本可能在支持的语言方面稍有差异。</em></p>
<p>因为 scripting 在早先 1.2 的时候出过安全问题,所以后来就都不再允许直接通过 POST 的内容里提交 scripting 代码了。现在有两种方式,一种是在 elasticsearch-1.4.1/config/ 目录下新建一个 scripts 目录,然后把准备要用的脚本都放在这个目录里,ES 会自动探测并加载编译;另一种是开启动态 scripting 功能,再通过 <code class="highlighter-rouge">/_script</code> 接口上传脚本。</p>
<p>下面示例两种实现获取 client_ip 字段的 C 段的统计的方式:</p>
<ol>
<li>通过简单的切割合并</li>
</ol>
<p>创建 <code class="highlighter-rouge">config/scripts/split.groovy</code> 文件,内容如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">doc</span><span class="o">[</span><span class="n">fieldname</span><span class="o">].</span><span class="na">value</span><span class="o">.</span><span class="na">split</span><span class="o">(</span><span class="sc">'.'</span><span class="o">)[</span><span class="mi">0</span><span class="o">..-</span><span class="mi">2</span><span class="o">].</span><span class="na">join</span><span class="o">(</span><span class="sc">'.'</span><span class="o">)</span>
</code></pre>
</div>
<p>稍等一下,看到 ES 的日志显示探测到并且编译成功后。就可以发送请求了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>curl '127.0.0.1:9200/logstash-2014.11.27/_search?pretty&size=0' -d '{
"aggs" : {
"ipaddr" : {
"terms" : {
"script" : "split",
"params" : {
"fieldname": "client_ip.raw"
}
}
}
}
}'
</code></pre>
</div>
<p><strong>注意这里一定要传递是 “not_analyzed” 的 字段过去!</strong> ES 流程上是先过分词器再到 scripting 模块的,这里要是切一下,到你脚本里就不知道长啥样了……</p>
<p>结果如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nt">"took"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">30</span><span class="p">,</span><span class="w">
</span><span class="nt">"timed_out"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nt">"_shards"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="p">,</span><span class="w">
</span><span class="nt">"successful"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="p">,</span><span class="w">
</span><span class="nt">"failed"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">786</span><span class="p">,</span><span class="w">
</span><span class="nt">"max_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mf">0.0</span><span class="p">,</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"aggregations"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"ipaddr"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"doc_count_error_upper_bound"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w">
</span><span class="nt">"sum_other_doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w">
</span><span class="nt">"buckets"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"127.0.0"</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_count"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">786</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
</div>
<ol>
<li>通过正则捕获</li>
</ol>
<p>前面的方式虽然达到目的,但是不像 splunk 的做法那么通用,所以更高级的是这样:</p>
<p>创建 <code class="highlighter-rouge">config/scripts/regex.groovy</code> 文件,内容如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">matcher</span> <span class="o">=</span> <span class="o">(</span> <span class="n">doc</span><span class="o">[</span><span class="n">fieldname</span><span class="o">].</span><span class="na">value</span> <span class="o">=~</span> <span class="o">/</span><span class="err">$</span><span class="o">{</span><span class="n">pattern</span><span class="o">}/</span> <span class="o">)</span>
<span class="k">if</span> <span class="o">(</span><span class="n">matcher</span><span class="o">.</span><span class="na">matches</span><span class="o">())</span> <span class="o">{</span>
<span class="n">matcher</span><span class="o">[</span><span class="mi">0</span><span class="o">][</span><span class="mi">1</span><span class="o">]</span>
<span class="o">}</span>
</code></pre>
</div>
<p>同样等识别编译,然后发送这样的请求:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>curl '127.0.0.1:9200/logstash-2014.11.27/_search?pretty&size=0' -d '{
"aggs" : {
"ipaddr" : {
"terms" : {
"script" : "regex",
"params" : {
"fieldname": "client_ip.raw",
"pattern": "^((?:\d{1,3}\.?){3})\.\d{1,3}$"
}
}
}
}
}'
</code></pre>
</div>
<p>得到一模一样的结果。</p>
<p>下一次试验一下在脚本中尝试加载其他库做更复杂处理的话,会如何呢?</p>
利用动态仪表板实现kibana单图表导出功能
2014-11-23T00:00:00+08:00
logstash
javascript
kibana
http://chenlinux.com/2014/11/23/kibana-panel-only-dashboard
<p>昨天和朋友聊天,说监控报表的话题,他们认为 kibana 的仪表板形式,还是偏重技术人员做监控的 screen 思路,对 erp 之类的报表不是很友好。要想跟其他系统结合,或者说嵌入到其他系统中,就必须得有单个图表的导出,或者 URL 引用方式。当时我直觉上的反应,就是这个没问题,可以通过 javascript 动态仪表板这个高级功能完成。回来试了一下,比我想的稍微复杂一点点,还是可以很轻松完成的。</p>
<p>读过<a href="http://kibana.logstash.es/content/dashboard-schema.html">仪表板纲要</a>一文,或者自己看过源代码中 <code class="highlighter-rouge">src/app/dashboards/logstash.json</code> 文件的人,应该都知道 kibana 中有些在页面配置界面里看不到的隐藏配置选项。其中很符合我们这次需求的,就有 <code class="highlighter-rouge">editable</code>, <code class="highlighter-rouge">collapsable</code> 等。所以,首先第一步,我们可以在自己的 <code class="highlighter-rouge">panel.js</code>(直接从 logstash.js 复制过来) 中,把这些关掉:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nx">dashboard</span><span class="p">.</span><span class="nx">rows</span> <span class="o">=</span> <span class="p">[</span>
<span class="p">{</span>
<span class="na">editable</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span> <span class="c1">//不显示每行的编辑按钮</span>
<span class="na">collapsable</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span> <span class="c1">//不显示每行的折叠按钮</span>
<span class="na">title</span><span class="p">:</span> <span class="s2">"Events"</span><span class="p">,</span>
<span class="na">height</span><span class="p">:</span> <span class="s2">"400px"</span><span class="p">,</span>
<span class="nx">panels</span> <span class="o">=</span> <span class="p">[{</span>
<span class="na">editable</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span> <span class="c1">//不显示面板的编辑按钮</span>
<span class="na">title</span><span class="p">:</span> <span class="s1">'events over time'</span><span class="p">,</span>
<span class="na">type</span><span class="p">:</span> <span class="s1">'histogram'</span><span class="p">,</span>
<span class="na">time_field</span><span class="p">:</span> <span class="nx">ARGS</span><span class="p">.</span><span class="nx">timefield</span><span class="o">||</span><span class="s2">"@timestamp"</span><span class="p">,</span>
<span class="na">auto_int</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="na">span</span><span class="p">:</span> <span class="mi">12</span>
<span class="p">}]</span>
<span class="p">}</span>
<span class="p">];</span>
<span class="nx">dashboard</span><span class="p">.</span><span class="nx">editable</span> <span class="o">=</span> <span class="kc">false</span><span class="p">;</span> <span class="c1">//不显示仪表板的编辑按钮</span>
<span class="nx">dashboard</span><span class="p">.</span><span class="nx">panel_hints</span> <span class="o">=</span> <span class="kc">false</span><span class="p">;</span> <span class="c1">//不显示面板的添加按钮</span>
</code></pre>
</div>
<p>然后要解决面板上方的 query 框和 filtering 框。这个同样在纲要介绍里说了,这两个特殊的面板是放在垂幕(pulldows)里的。所以,直接关掉垂幕就好了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nx">dashboard</span><span class="p">.</span><span class="nx">pulldowns</span> <span class="o">=</span> <span class="p">[];</span>
</code></pre>
</div>
<p>然后再往上是顶部栏。顶部栏里有时间选择器,这个跟垂幕一样是可以关掉的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nx">dashboard</span><span class="p">.</span><span class="nx">nav</span> <span class="o">=</span> <span class="p">[];</span>
</code></pre>
</div>
<p>好了,javascript 里可以关掉的,都已经关了。</p>
<p>但是运行起来,发现顶部栏里虽然是没有时间选择器和配置编辑按钮了,本身这个黑色条带和 logo 图什么的,却依然存在!这时候我想起来有时候 config.js 没写对,<code class="highlighter-rouge">/_nodes</code> 获取失败的时候,打开的页面就是背景色外加这个顶条 —— 也就是说,这部分代码是写在 <code class="highlighter-rouge">index.html</code> 里的,不受 <code class="highlighter-rouge">app/dashboards/panel.js</code> 控制。</p>
<p>所以这里就得去修改一下 <code class="highlighter-rouge">index.html</code> 了。不过为了保持兼容性,我这里没有直接删除顶部栏的代码,而是用了 angularjs 中很常用的 <code class="highlighter-rouge">ng-show</code> 指令:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nt"><div</span> <span class="na">ng-cloak</span> <span class="na">class=</span><span class="s">"navbar navbar-static-top"</span> <span class="na">ng-show=</span><span class="s">"dashboard.current.nav.length"</span><span class="nt">></span>
</code></pre>
</div>
<p>因为之前关闭时间选择器的时候,已经把这个 nav 数组定义为空了,所以只要判断一下数组长度即可。</p>
<p>效果如下:</p>
<p><img src="http://ww1.sinaimg.cn/large/3dbd9afagw1eml6f9xqltj20lc0fuwfu.jpg" alt="single panel" /></p>
<p>因为 <code class="highlighter-rouge">dashboard.services</code> 的定义没有做修改,所以这个其实照样支持你用鼠标拉动选择时间范围,支持你在 URL 后面加上 <code class="highlighter-rouge">?query=status:404&from=1h</code> 这样的参数,效果都是对的。只不过不会再让你看到这些文字显示在页面上了。</p>
<p>如果要求再高一点,其实完全可以在 <code class="highlighter-rouge">ARGS</code> 里处理更复杂的参数,比如直接 <code class="highlighter-rouge">?type=terms&field=host&value_field=requesttime</code> 就生成 <code class="highlighter-rouge">dashboard.rows[0].panels[0]</code> 里的对应参数,达到自动控制图表类型和效果的目的。</p>
用 phantomjs 截图
2014-11-20T00:00:00+08:00
javascript
kibana
http://chenlinux.com/2014/11/20/phantomjs-snapshot
<p>昨儿给 kibana 加上了 table 面板数据导出成 CSV 的功能。朋友们就问了,那其他面板的图表怎么导出保存呢?其实直接截图就好了嘛……</p>
<p>FireFox 有插件用来截全网页图。不过如果作为定期的工作,这么搞还是比较麻烦的,需要脚本化下来。这时候就可以用上 phantomjs 软件了。phantomjs 是一个基于 webkit 引擎做的 js 脚本库。可以通过 js 程序操作 webkit 浏览器引擎,实现各种浏览器功能。</p>
<p>因为用了 webkit ,所以软件编译起来挺麻烦的,建议是直接从官方下载二进制包用得了。</p>
<p>想要给 kibana 页面截图,几行代码就够了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="kd">var</span> <span class="nx">page</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'webpage'</span><span class="p">).</span><span class="nx">create</span><span class="p">();</span>
<span class="kd">var</span> <span class="nx">address</span> <span class="o">=</span> <span class="s1">'http://kibana.dip.sina.com.cn/#/dashboard/elasticsearch/h5_view'</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">output</span> <span class="o">=</span> <span class="s1">'kibana.png'</span><span class="p">;</span>
<span class="nx">page</span><span class="p">.</span><span class="nx">viewportSize</span> <span class="o">=</span> <span class="p">{</span> <span class="na">width</span><span class="p">:</span> <span class="mi">1366</span><span class="p">,</span> <span class="na">height</span><span class="p">:</span> <span class="mi">600</span> <span class="p">};</span>
<span class="nx">page</span><span class="p">.</span><span class="nx">open</span><span class="p">(</span><span class="nx">address</span><span class="p">,</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">status</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">status</span> <span class="o">!==</span> <span class="s1">'success'</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'Unable to load the address!'</span><span class="p">);</span>
<span class="nx">phantom</span><span class="p">.</span><span class="nx">exit</span><span class="p">();</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nb">window</span><span class="p">.</span><span class="nx">setTimeout</span><span class="p">(</span><span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="nx">page</span><span class="p">.</span><span class="nx">render</span><span class="p">(</span><span class="nx">output</span><span class="p">);</span>
<span class="nx">phantom</span><span class="p">.</span><span class="nx">exit</span><span class="p">();</span>
<span class="p">},</span> <span class="mi">20000</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">});</span>
</code></pre>
</div>
<p>这里两个要点:</p>
<ol>
<li>要设置 <code class="highlighter-rouge">viewportSize</code> 里的宽度,否则效果会变成单个 panel 依次往下排列。</li>
<li>要设置 <code class="highlighter-rouge">setTimeout</code>,否则在获取完 index.html 后就直接返回了,只能看到一个大白板。用 phantomjs 截取 angularjs 这类单页 MVC 框架应用时一定要设置这个。</li>
</ol>
在 kibana 里实现去重计数
2014-11-19T00:00:00+08:00
logstash
kibana
elasticsearch
javascript
http://chenlinux.com/2014/11/19/uniq-count-kibana
<p>如何在 elk 里统计或者展示去重计数,是一个持续很久的需求了。几乎每个月都会有新手提问题说:“我怎么在 kibana 里统计网站 UV 啊?”可惜这个问题的回答总是:做不到……</p>
<p>其实 Elasticsearch 从 1.1.0 版本开始已经可以做到<a href="http://www.elasticsearch.org/blog/count-elasticsearch/">去重统计</a>了。但是 kibana3 本身是在 0.90 版本基础上实现的,所以也就没办法了。</p>
<p>今天抽出时间,把 histogram 面板的代码重写了一遍,用 aggregations 接口替换了 facets 接口。改造完成后,再加上去重就很容易了。</p>
<p>aggregations 接口最大的特点是层级关系。不过也不是可以完全随便嵌套的,原先 date_histogram facets 里的 global 参数,被拆分成了 global aggregation,但是这个 global aggregation 就强制要求必须用在顶层。所以最后 request 相关代码就变成了这个样子:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="kd">var</span> <span class="nx">aggr</span> <span class="o">=</span> <span class="nx">$scope</span><span class="p">.</span><span class="nx">ejs</span><span class="p">.</span><span class="nx">DateHistogramAggregation</span><span class="p">(</span><span class="nx">q</span><span class="p">.</span><span class="nx">id</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">mode</span> <span class="o">===</span> <span class="s1">'count'</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">aggr</span> <span class="o">=</span> <span class="nx">aggr</span><span class="p">.</span><span class="nx">field</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">time_field</span><span class="p">);</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">mode</span> <span class="o">===</span> <span class="s1">'uniq'</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">aggr</span> <span class="o">=</span> <span class="nx">aggr</span><span class="p">.</span><span class="nx">field</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">time_field</span><span class="p">).</span><span class="nx">agg</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">ejs</span><span class="p">.</span><span class="nx">CardinalityAggregation</span><span class="p">(</span><span class="nx">q</span><span class="p">.</span><span class="nx">id</span><span class="p">).</span><span class="nx">field</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">value_field</span><span class="p">));</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">aggr</span> <span class="o">=</span> <span class="nx">aggr</span><span class="p">.</span><span class="nx">field</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">time_field</span><span class="p">).</span><span class="nx">agg</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">ejs</span><span class="p">.</span><span class="nx">StatsAggregation</span><span class="p">(</span><span class="nx">q</span><span class="p">.</span><span class="nx">id</span><span class="p">).</span><span class="nx">field</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">value_field</span><span class="p">));</span>
<span class="p">}</span>
<span class="nx">request</span> <span class="o">=</span> <span class="nx">request</span><span class="p">.</span><span class="nx">agg</span><span class="p">(</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">ejs</span><span class="p">.</span><span class="nx">GlobalAggregation</span><span class="p">(</span><span class="nx">q</span><span class="p">.</span><span class="nx">id</span><span class="p">).</span><span class="nx">agg</span><span class="p">(</span>
<span class="nx">$scope</span><span class="p">.</span><span class="nx">ejs</span><span class="p">.</span><span class="nx">FilterAggregation</span><span class="p">(</span><span class="nx">q</span><span class="p">.</span><span class="nx">id</span><span class="p">).</span><span class="nx">filter</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">ejs</span><span class="p">.</span><span class="nx">QueryFilter</span><span class="p">(</span><span class="nx">query</span><span class="p">)).</span><span class="nx">agg</span><span class="p">(</span>
<span class="nx">aggr</span><span class="p">.</span><span class="nx">interval</span><span class="p">(</span><span class="nx">_interval</span><span class="p">)</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="p">).</span><span class="nx">size</span><span class="p">(</span><span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">annotate</span><span class="p">.</span><span class="nx">enable</span> <span class="p">?</span> <span class="nx">$scope</span><span class="p">.</span><span class="nx">panel</span><span class="p">.</span><span class="nx">annotate</span><span class="p">.</span><span class="nx">size</span> <span class="p">:</span> <span class="mi">0</span><span class="p">);</span>
</code></pre>
</div>
<p>完整的代码已经提交到 github,见 <a href="https://github.com/chenryn/kibana-authorization/commit/6cb4d28a6c610d28680fffdb81c9f6c83cfaf488">https://github.com/chenryn/kibana-authorization/commit/6cb4d28a6c610d28680fffdb81c9f6c83cfaf488</a></p>
【翻译】Kibana 4 beta 2 发布
2014-11-18T00:00:00+08:00
logstash
kibana
http://chenlinux.com/2014/11/18/kibana4-beta-2-get-now
<p>原文地址见:<a href="http://www.elasticsearch.org/blog/kibana-4-beta-2-get-now/">http://www.elasticsearch.org/blog/kibana-4-beta-2-get-now/</a></p>
<hr />
<p>哈哈哈哈哈哈哈哈哈!来啦!Kibana 4 Beta 2 现在正式雪地 360° 裸跪求调戏,包括你家喵星人都行,只要你给反馈。(译者注:ES 的发版日志越来越活泼,我也翻译的更中文化点好了)</p>
<p>如果你已经等不及要开动,从<a href="http://www.elasticsearch.org/overview/kibana/installation/">这里</a>下载 Kibana 4 Beta 2,否则继续阅读下面的亮点。</p>
<p>除了很多小的修复和改进,这个版本里还有一些非常值得一看的新东西:</p>
<h2 id="section">地图支持</h2>
<p>地图回来啦,而且比过去更强大了!新的瓦片式地图可视化用上了 Elasticsearch 强大的 <code class="highlighter-rouge">geohash_grid</code> 来显示地理数据,比如可视化展示相对响应时间:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/11/Screen-Shot-2014-11-10-at-3.20.00-PM-1024x547.png" alt="map" /></p>
<h2 id="section-1">可视化选项</h2>
<p>在 Beta 1 里,柱状图是固定成堆叠式的。在 Kibana 4 Beta 2 里,我们添加了选项让你修改可视化展示数据的方式。比如,分组柱状图:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/11/Screen-Shot-2014-11-10-at-3.15.56-PM-1024x564.png" alt="grouped bars" /></p>
<p>或者百分比式柱状图:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/11/Screen-Shot-2014-11-10-at-4.02.57-PM-1024x537.png" alt="Percent bars" /></p>
<h2 id="section-2">区域图</h2>
<p>Beta 2 里区域图也回来了,包括堆叠式和非堆叠式:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/11/Screen-Shot-2014-11-10-at-3.13.21-PM-1024x564.png" alt="area" /></p>
<h2 id="section-3">高级参数</h2>
<p>我们目标是支持尽可能多的 Elasticsearch 特性,不过有时候我们确实还没覆盖到某个聚合选项,而你偏偏现在就要用它。这种情况下,我们引入了 JSON 输入,让你可以定义附加的聚合参数到发送的请求里。比如,你可能想在一个 <code class="highlighter-rouge">terms</code> 聚合里传递一个 <code class="highlighter-rouge">shard_size</code>,或者在一个基数聚合里调大 <code class="highlighter-rouge">precision_threshold</code>。在下面示例中,我们传了一个小脚本作为高级参数,计算 <code class="highlighter-rouge">bytes</code> 字段的 <code class="highlighter-rouge">_value</code> 的对数值,然后用它作为 X 轴:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/11/Screen-Shot-2014-11-10-at-3.41.13-PM-1024x538.png" alt="scripts" /></p>
<h2 id="section-4">数据表格</h2>
<p>有时候你想要个动态图,有时候可能只想要数值就够了。数据表格可视化达成你这个愿望:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/11/Screen-Shot-2014-11-10-at-3.45.02-PM-1024x536.png" alt="data table" /></p>
<h2 id="section-5">喂!我的仪表盘哪去了?</h2>
<p>Kibana 内部使用的索引从 <code class="highlighter-rouge">kibana-int</code> 改名叫 <code class="highlighter-rouge">.kibana</code> 了。我们建议你从老索引里把文档(比如:仪表盘,设置,可视化等)都挪到新索引来。不过,你还是可以在 kibana.yml 里直接定义 <code class="highlighter-rouge">kibanaIndex: "kibana-int"</code> 的。</p>
<h2 id="section-6">我们现在在做什么?</h2>
<p>可以从 <a href="https://github.com/elasticsearch/kibana/labels/roadmap">roadmap</a> 上看到我们离 Kibana 4 正式版还有多远。另外,我们永远欢迎你在 <a href="https://github.com/elasticsearch/kibana/issues">GitHub</a> 的反馈、bug 报告、补丁等等。</p>
用 perl6-bench 做 perl6 性能对比
2014-10-28T00:00:00+08:00
perl
http://chenlinux.com/2014/10/28/perl6-bench
<p>Perl6 成员上周在奥地利大会上做了一次大聚集,写了不少博客讲过去几个月的优化以及未来几个月的优化。但是我发现似乎从8月以来就一直没有正式的 perl6-bench 的图表报告了。于是想:干脆自己跑一把吧。</p>
<p>perl6-bench 项目地址见:<a href="https://github.com/japhb/perl6-bench">https://github.com/japhb/perl6-bench</a>。</p>
<p>项目的主程序 <code class="highlighter-rouge">bench</code> 本身是用 Perl6 写的。所以运行前,得先安装好 Rakudo Star:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget http://rakudo.org/downloads/star/rakudo-star-2014.09.tar.gz
tar zxvf rakudo-star-2014.09.tar.gz
cd rakudo-star-2014.09
perl Configure.pl --backend=moar --gen-moar
</code></pre>
</div>
<p>编译完成后,会在 rakudo-star 目录下创建一个 <code class="highlighter-rouge">install</code> 子目录,里面有 <code class="highlighter-rouge">bin</code>,<code class="highlighter-rouge">lib</code> 等编译完成的文件,把这个 bin 加入到你的 $PATH 里去。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sed -i 's!\(PATH=.*\)$!\1:~/download/rakudo-star-2014.09/install/bin!' ~/.bash_profile
source ~/.bash_profile
</code></pre>
</div>
<p>项目的测试程序 <code class="highlighter-rouge">timeall</code> 是用 Perl5 写的。运行前,也得安装几个 CPAN 模块:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>cpanm Capture::Tiny Data::Alias DateTime JSON JSON::XS List::MoreUtils IPC::Run
</code></pre>
</div>
<p>然后就可以开始测试了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>./bench setup
</code></pre>
</div>
<p>这个命令会在 <code class="highlighter-rouge">components</code> 子目录下逐一 clone 下来各种可以测试的 perl6 实现的源代码 git 库,包括有:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>arane niecza nqp-jvm nqp-parrot perl5 rakudo-jvm rakudo-parrot
moarvm nqp-js nqp-moar parrot perlito rakudo-moar
</code></pre>
</div>
<p>下面就开始正式测试了。用时同样会比较长,和上面 git clone 一样,都建议放在 screen 里运行。</p>
<p>然后设定本次测试你打算对比哪些:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>export CHECKOUTS='perl5/v5.20.1 rakudo-jvm/2014.10 rakudo-moar/2014.10 rakudo-moar/2014.09'
</code></pre>
</div>
<p>这个写法规范是:git 库名/git tag名</p>
<p>然后运行:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>./bench extract $CHECKOUTS
</code></pre>
</div>
<p>这一步会分别 checkout 具体的 tag 到同级的新目录里,然后开始编译:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>./bench build $CHECKOUTS
</code></pre>
</div>
<p>然后运行测试程序:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>./bench time $CHECKOUTS
</code></pre>
</div>
<p>一共有 65 个测试,测试项在 <code class="highlighter-rouge">microbenchworks.pl</code> 文件的大数组里定义了。</p>
<p>我在测试中发现,第 15/65 测试用例,在 <code class="highlighter-rouge">nqp-moar</code> 时会死循环运行,无法正常完成测试,已回报给作者。</p>
<p>./bench 还可以添加其他运行参数。比如 <code class="highlighter-rouge">./bech --verbose time $CHECKOUTS</code>。注意参数必须写在 “time” 前面。这是 Perl6 的 MAIN 函数特性:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>multi MAIN ('time', *@components, :$variants?, :$tests?, :$tests-tagged?,
:$runs?, :$enough-time?, :$min-scaling-points?,
Bool :$verbose?) { }
</code></pre>
</div>
<p>代码里用了 <code class="highlighter-rouge">*@components</code>,所有写在 “time” 后面的参数都会存入这个数组。</p>
<p>最后运行结果对比评分:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>./bench compare $CHECKOUTS
</code></pre>
</div>
<p>结果显示,moar 比 jvm 领先一些,比 perl5 还差着呢:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>==> perl6-bench version 997c920 (ignoring startup time and compile time)
--- showing PEAK RATE (/s), TIMES SLOWER THAN FASTEST (x), and SUMMARY SCORES (skipping incomplete data)
- Perl 5 - -------------- Perl 6 --------------
v5.20.1 2014.10 2014.09 2014.10
perl5 rakudo rakudo rakudo
TEST perl5 jvm moarvm moarvm
-------------------------------------------------
empty -- 0/s 4/s 4/s
FAIL 34.0x 1.0x 1.1x
zero -- 0/s 4/s 4/s
FAIL 33.3x 1.0x 1.1x
hello -- 0/s 4/s 4/s
FAIL 33.2x 1.0x 1.1x
while_empty 26678545/s 223006/s 1730328/s 3403743/s
1.0x 119.6x 15.4x 7.8x
while_empty_native 26800035/s 1291144447/s 27583644/s 168949423/s
48.2x 1.0x 46.8x 7.6x
while_bind -- 249216/s 1682441/s 3381083/s
FAIL 13.6x 2.0x 1.0x
while_concat 13404147/s 26589/s 166714/s 206047/s
1.0x 504.1x 80.4x 65.1x
while_concat_native 13400671/s 65891/s 4138382/s 5216637/s
1.0x 203.4x 3.2x 2.6x
while_int2str 6026835/s 57112/s 364208/s 455797/s
1.0x 105.5x 16.5x 13.2x
while_int2str_native 6283498/s 111754/s 543142/s 671402/s
1.0x 56.2x 11.6x 9.4x
while_int2str_concat 8711901/s 7006/s 89566/s 93480/s
1.0x 1243.5x 97.3x 93.2x
while_int2str_concat_native 8403097/s 13824/s 153347/s 167585/s
1.0x 607.9x 54.8x 50.1x
while_push_join 3656434/s 15223/s 18917/s 111952/s
1.0x 240.2x 193.3x 32.7x
while_push 7821809/s 90685/s 21289/s 239678/s
1.0x 86.3x 367.4x 32.6x
while_pushme 14440088/s 3184098/s 1225845/s 1560029/s
1.0x 4.5x 11.8x 9.3x
while_array_set 6171761/s 112655/s 276032/s 335751/s
1.0x 54.8x 22.4x 18.4x
while_hash_set 1525235/s 58647/s 158810/s 171691/s
1.0x 26.0x 9.6x 8.9x
postwhile_nil 36412794/s 515093/s 2939870/s 4147168/s
1.0x 70.7x 12.4x 8.8x
postwhile_nil_native 36083908/s 1676476937/s 34716639/s 167547820/s
46.5x 1.0x 48.3x 10.0x
loop_empty 24051967/s 257307/s 1686547/s 3321511/s
1.0x 93.5x 14.3x 7.2x
loop_empty_native 24181034/s 2276716196/s 28050857/s 193967640/s
94.2x 1.0x 81.2x 11.7x
for_empty 33943008/s 894886/s 2315939/s 2515590/s
1.0x 37.9x 14.7x 13.5x
for_bind -- 1571035/s 2331450/s 2586230/s
FAIL 1.6x 1.1x 1.0x
for_assign 17713024/s 1532922/s 2006784/s 2391570/s
1.0x 11.6x 8.8x 7.4x
for_assign_native 17765094/s 1658168/s 1895988/s 2006162/s
1.0x 10.7x 9.4x 8.9x
for_postinc 16640609/s 386218/s 1398445/s 1802886/s
1.0x 43.1x 11.9x 9.2x
for_postinc_native 16670507/s 1037555/s 1859233/s 1994065/s
1.0x 16.1x 9.0x 8.4x
for_concat 14998496/s 29144/s 182410/s 205988/s
1.0x 514.6x 82.2x 72.8x
for_concat_native 15053529/s 49506/s 1353377/s 1465293/s
1.0x 304.1x 11.1x 10.3x
for_concat_2 8646049/s 15854/s 107213/s 117943/s
1.0x 545.4x 80.6x 73.3x
for_concat_2_native 8659225/s 23751/s 791213/s 986208/s
1.0x 364.6x 10.9x 8.8x
for_push 8496867/s 122034/s 25166/s 333166/s
1.0x 69.6x 337.6x 25.5x
for_array_set 7810807/s 57463/s 286036/s 388650/s
1.0x 135.9x 27.3x 20.1x
for_hash_set 1567864/s 32265/s 168643/s 171446/s
1.0x 48.6x 9.3x 9.1x
reduce_range 4964114/s 181283/s 318258/s 345797/s
1.0x 27.4x 15.6x 14.4x
reduce_int_comb_range 470778/s 1495/s 3355/s 3406/s
1.0x 314.8x 140.3x 138.2x
any_equals 2646212/s 15684/s 61867/s 81787/s
1.0x 168.7x 42.8x 32.4x
trim_string 13660958/s 33565139/s 9291330/s 17910365/s
2.5x 1.0x 3.6x 1.9x
split_string_constant 5615519/s 100014/s 133572/s 171231/s
1.0x 56.1x 42.0x 32.8x
split_string_regex 2017912/s 4137/s 12573/s 16553/s
1.0x 487.8x 160.5x 121.9x
charrange 363103/s 3416/s 19831/s 24667/s
1.0x 106.3x 18.3x 14.7x
charrange_ignorecase 363529/s 3788/s 14433/s 17899/s
1.0x 96.0x 25.2x 20.3x
visit_2d_indices_while 7276084/s 152635/s 746903/s 1484712/s
1.0x 47.7x 9.7x 4.9x
visit_2d_indices_while_native 11180261/s 553619/s 1177498/s 1451682/s
1.0x 20.2x 9.5x 7.7x
visit_2d_indices_loop 10123295/s 177783/s 834515/s 1843586/s
1.0x 56.9x 12.1x 5.5x
visit_2d_indices_loop_native 12457926/s 440172780/s 1227550/s 1431680/s
35.3x 1.0x 358.6x 307.5x
visit_2d_indices_for 8548538/s 255887/s 675743/s 847728/s
1.0x 33.4x 12.7x 10.1x
visit_2d_indices_cross 1367865/s 4685/s 31407/s 40470/s
1.0x 292.0x 43.6x 33.8x
create_and_copy_2d_grid_cross 541914/s 2230/s 11564/s 13778/s
1.0x 243.0x 46.9x 39.3x
create_and_iterate_hash_kv -- 1564/s 12248/s 12651/s
FAIL 8.1x 1.0x 1.0x
rat_mul_div_cancel 7439/s 4852/s 33910/s 40614/s
5.5x 8.4x 1.2x 1.0x
rat_harmonic 1080/s 1732/s 11089/s 11678/s
10.8x 6.7x 1.1x 1.0x
rand 10885068/s 230938/s 183511/s 213786/s
1.0x 47.1x 59.3x 50.9x
array_set_xx 13585287/s 1533694/s 545243/s 597926/s
1.0x 8.9x 24.9x 22.7x
parse-json 23/s 1/s 1/s 1/s
1.0x 29.6x 37.0x 22.8x
parse-json-no-obj-creation -- 1/s 1/s 1/s
SKIP 1.4x 1.9x 1.0x
rc-forest-fire 1374/s 2/s 9/s 9/s
1.0x 588.4x 155.8x 146.7x
rc-man-or-boy-test 187464/s -- 41252/s 39966/s
1.0x FAIL 4.5x 4.7x
rc-self-describing-numbers 219156/s 571/s 775/s 782/s
1.0x 383.9x 282.9x 280.1x
rc-dragon-curve 149131/s 1704/s 5937/s 6260/s
1.0x 87.5x 25.1x 23.8x
rc-9-billion-names 1821/s 93/s 216/s 500/s
1.0x 19.7x 8.4x 3.6x
rc-mandelbrot 1168/s 702/s 1440/s 1519/s
1.3x 2.2x 1.1x 1.0x
spinner 971/s 4/s 5/s 5/s
1.0x 228.7x 193.1x 182.9x
rc-forest-fire-stringify 11162/s 25/s 35/s 41/s
1.0x 438.8x 314.7x 275.2x
string-escape 1448636/s -- -- --
1.0x FAIL FAIL FAIL
=================================================
SUMMARY SCORE 2253.9 40.5 100.0 139.6
</code></pre>
</div>
<p>如何把数据用图的形式展示,我还没有找到办法。</p>
ESCC 参会笔记
2014-10-27T00:00:00+08:00
elasticsearch
kibana
http://chenlinux.com/2014/10/27/escc
<p>10 月 25 号举办了 ESCC(ElasticSearch China Conference)。作为个人习惯,稍作记录。</p>
<p>会议的筹备时间其实特别短,8 月 20 号,ansj 说他跟 medcl 喝了瓶芬达,然后就敲定开搞。中间 medcl 默默承担了各种工作,直到 9 月 11 号,告诉我们场地已经搞定,分头写 ppt 吧。然后 9 月 30 号在 meetup.com 上创建组织,正式发出 meetup 公告。</p>
<p>medcl 办事是放心的。两年前,我们二三十号人在人人公司培训教室里听 medcl 一个人培训式的宣讲了一下午,场控能力绝对有保障。</p>
<p>最后我几乎是卡着点到的会场。国奥花园酒店非常贴心,一路上主动贴好了指路牌,而且免费给会场内每个座位发了一瓶矿泉水。会议中,甚至有服务员悄悄给前排的茶杯里续水……</p>
<p>ansj 第一个演讲,话题很学术,演讲很生动。演讲结束后提问者络绎不绝,眼瞧着赞助商的小礼品都要不够发了,逼得 medcl 站出来表示后续问题转成私下讨论。</p>
<p>我第二个演讲,时间上把握得还算好。由于转换成 ppt 在 windows 下播放,所以准备的 demo 就没在演讲中使用。不过在后来 QA 环节,有妹纸问到 Kibana3 和 Kibana4 的区别的时候,完全应该换成自己电脑演示一下这两个版本完全不同的效果的。感觉这个问题我纯靠口述基本没能让听众明白……</p>
<p>而后祝威廉告诉我:因为讲台灯光问题,ppt 拍摄效果很糟糕。等我自己回座位看后面的演讲,黑底白字确实比白底黑字效果好多了。这点算是教训,以后要注意。</p>
<p>接下来演讲的是黄琛。之前只看过他开源的 repo 的 README 说明,这次还见到了完整的 demo 页。漂亮的自定义语法,我个人想:如果能再实现一层,<code class="highlighter-rouge">| timechart</code>/<code class="highlighter-rouge">| pie</code> 这样的语法接在现有语法的后面,就能自动在页面生成时间序列图,饼图之类的,那就更帅了!会场上好几位同事问我为什么要搞这么个语法,感觉即不像搜索也不像 SQL。答曰:splunk 用户看着亲切~</p>
<p>然后是祝威廉。演讲主题是自己的一套 CS 系统。很高兴能在 ESCC 才刚第三届的时候,就有这种非强相关的话题,我觉得这是社区活跃的一种象征。听完演讲,我的感觉是:CS 主要点在那层 Strategies 上。由 strategies 来完成客户端接收,数据存储的请求,以及实际的逻辑处理。或许可以类比数据库的中间件?这种解耦确实在规模运维和频繁变更的时候更方便了。不过 ES 是一个讲究上手简单的系统,应该不太会走这个路线。</p>
<p>最后是刘刚。回归 ES 功能本身,讲了一下 <code class="highlighter-rouge">func_score</code>。跟我工作不太相干,之前了解也不多。只能表示很好很强大了……</p>
<p>总的来说,5 个演讲分别涉及 ES 的分词原理,聚合函数用法,请求解析的内部实现,服务架构的对比以及评分的用法。应该是比较全面了。(当然,我个人觉得其实 medcl 没时间也可以把半个月前在 Qcon shanghai 的演讲在北京同好这再讲一次)</p>
<p>会议休息和结束后,大家都在通过微信互加好友(我 ppt 上的微博二维码估计是没人扫描了)。</p>
<p>不过直到我们七个人一起晚饭的时候,才想起来为啥不统一在会场搞一个“面对面建群”,把大家都加进一个统一的群里呢?</p>
<p>吃饭的时候互叙了一轮年庚,发现我们多是 85 后。ES 社区也算是年轻社区了。刷微博看到有用户评价说:“收获满满,哈哈。对elasticsearch 中文社区好感直线上升,这是一个不吹水的社区。”开心!</p>
<p>Anyway,感谢这 150 多名冒着严重雾霾来聚会的 Elasticsearch 爱好者。</p>
<p><em>注:后来发现其实在半个月前,我在 perlweekly 里就发过一篇 ES 的自定义打分的博客链接:<a href="http://blogs.perl.org/users/mateu/2014/10/elasticsearch-custom-scoring.html">http://blogs.perl.org/users/mateu/2014/10/elasticsearch-custom-scoring.html</a></em></p>
在终端命令行上调试 grok 表达式
2014-10-19T07:43:00+08:00
logstash
ruby
http://chenlinux.com/2014/10/19/grokdebug-commandline
<p>用 logstash 的人都知道在 <a href="http://grokdebug.herokuapp.com">http://grokdebug.herokuapp.com</a> 上面调试 grok 正则表达式。现在问题来了:<del>翻墙技术哪家强?</del> 页面中用到了来自 google 域名的 js 文件,所以访问经常性失败。所以,在终端上通过命令行方式快速调试成了必需品。</p>
<p>其实在 logstash 还在 1.1 的年代的时候,官方 wiki 上是有一批专门教大家怎么通过 irb 交互式测试 grok 表达式的。但不知道为什么后来 wiki 这页没了…… 好在代码本身不复杂,稍微写几行脚本,就可以达到目的了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/env ruby</span>
<span class="nb">require</span> <span class="s1">'rubygems'</span>
<span class="n">gem</span> <span class="s1">'jls-grok'</span><span class="p">,</span> <span class="s1">'=0.11.0'</span>
<span class="nb">require</span> <span class="s1">'grok-pure'</span>
<span class="nb">require</span> <span class="s1">'optparse'</span>
<span class="nb">require</span> <span class="s1">'ap'</span>
<span class="n">options</span> <span class="o">=</span> <span class="p">{}</span>
<span class="no">ARGV</span><span class="p">.</span><span class="nf">push</span><span class="p">(</span><span class="s1">'-h'</span><span class="p">)</span> <span class="k">if</span> <span class="no">ARGV</span><span class="p">.</span><span class="nf">size</span> <span class="o">===</span> <span class="mi">0</span>
<span class="no">OptionParser</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span> <span class="o">|</span><span class="n">opts</span><span class="o">|</span>
<span class="n">opts</span><span class="p">.</span><span class="nf">banner</span> <span class="o">=</span> <span class="s1">'Run grokdebug at your terminal.'</span>
<span class="n">options</span><span class="p">[</span><span class="ss">:dirs</span><span class="p">]</span> <span class="o">=</span> <span class="sx">%w(patterns)</span>
<span class="n">options</span><span class="p">[</span><span class="ss">:named</span><span class="p">]</span> <span class="o">=</span> <span class="kp">false</span>
<span class="n">opts</span><span class="p">.</span><span class="nf">on</span><span class="p">(</span><span class="s1">'-d DIR1,DIR2'</span><span class="p">,</span> <span class="s1">'--dirs DIR1,DIR2'</span><span class="p">,</span> <span class="no">Array</span><span class="p">,</span> <span class="s1">'Set grok patterns directories. Default: "./patterns"'</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">value</span><span class="o">|</span>
<span class="n">options</span><span class="p">[</span><span class="ss">:dirs</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>
<span class="k">end</span>
<span class="n">opts</span><span class="p">.</span><span class="nf">on</span><span class="p">(</span><span class="s1">'-m MESSAGE'</span><span class="p">,</span> <span class="s1">'--msg MESSAGE'</span><span class="p">,</span> <span class="s1">'Your raw message to be matched'</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">value</span><span class="o">|</span>
<span class="n">options</span><span class="p">[</span><span class="ss">:message</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>
<span class="k">end</span>
<span class="n">opts</span><span class="p">.</span><span class="nf">on</span><span class="p">(</span><span class="s1">'-p PATTERN'</span><span class="p">,</span> <span class="s1">'--pattern PATTERN'</span><span class="p">,</span> <span class="s1">'Your grok pattern to be compiled'</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">value</span><span class="o">|</span>
<span class="n">options</span><span class="p">[</span><span class="ss">:pattern</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>
<span class="k">end</span>
<span class="n">opts</span><span class="p">.</span><span class="nf">on</span><span class="p">(</span><span class="s1">'-n'</span><span class="p">,</span> <span class="s1">'--named'</span><span class="p">,</span> <span class="s1">'Named captures only'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">options</span><span class="p">[</span><span class="ss">:named</span><span class="p">]</span> <span class="o">=</span> <span class="kp">true</span>
<span class="k">end</span>
<span class="k">end</span><span class="p">.</span><span class="nf">parse!</span>
<span class="n">grok</span> <span class="o">=</span> <span class="no">Grok</span><span class="p">.</span><span class="nf">new</span>
<span class="n">options</span><span class="p">[</span><span class="ss">:dirs</span><span class="p">].</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">dir</span><span class="o">|</span>
<span class="k">if</span> <span class="no">File</span><span class="p">.</span><span class="nf">directory?</span><span class="p">(</span><span class="n">dir</span><span class="p">)</span>
<span class="n">dir</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">dir</span><span class="p">,</span> <span class="s2">"*"</span><span class="p">)</span>
<span class="k">end</span>
<span class="no">Dir</span><span class="p">.</span><span class="nf">glob</span><span class="p">(</span><span class="n">dir</span><span class="p">).</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">file</span><span class="o">|</span>
<span class="n">grok</span><span class="p">.</span><span class="nf">add_patterns_from_file</span><span class="p">(</span><span class="n">file</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">grok</span><span class="p">.</span><span class="nf">compile</span><span class="p">(</span><span class="n">options</span><span class="p">[</span><span class="ss">:pattern</span><span class="p">],</span> <span class="n">options</span><span class="p">[</span><span class="ss">:named</span><span class="p">])</span>
<span class="n">ap</span> <span class="n">grok</span><span class="p">.</span><span class="nf">match</span><span class="p">(</span><span class="n">options</span><span class="p">[</span><span class="ss">:message</span><span class="p">]).</span><span class="nf">captures</span><span class="p">()</span>
</code></pre>
</div>
<p>测试一下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ sudo gem install jls-grok awesome_print
$ ruby grokdebug.rb
Run grokdebug at your terminal.
-d, --dirs DIR1,DIR2 Set grok patterns directories. Default: "./patterns"
-m, --msg MESSAGE Your raw message to be matched
-p, --pattern PATTERN Your grok pattern to be compiled
-n, --named Named captures only
$ ruby grokdebug.rb -m 'abc123' -p '%{NUMBER:test}'
{
"test" => [
[0] "123"
],
"BASE10NUM" => [
[0] "123"
]
}
$ ruby grokdebug.rb -m 'abc123' -p '%{NUMBER:test:float}' -n
{
"test" => [
[0] 123.0
]
}
</code></pre>
</div>
<p>没错,我这比 grokdebug 网站还多了类型转换的功能。它用的 jls-grok 是 0.10.10 版,而我用的是最新的 0.11.0 版。</p>
Rsyslog 性能数据 impstats 直接写入 Elasticsearch
2014-10-19T05:48:00+08:00
logstash
rsyslog
elasticsearch
http://chenlinux.com/2014/10/19/rsyslog-impstats-elasticsearch
<p>Rsyslog 的性能数据,可以通过自带的 impstats 插件输出。但是在用的比较复杂的场景下,每次输出都会有好几十个 action 的各种状态,肉眼观察变得比较困难,这时候,我们可以直接输出给 Elasticsearch ,然后利用 Kibana 做快速搜索和分析。</p>
<p>Rsyslog 官方提供了直接输出给 Elasticsearch 的插件:<code class="highlighter-rouge">omelasticsearch</code>。配置如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>module(load="omelasticsearch")
module(load="impstats" interval="120" severity="6" log.syslog="on" format="json" resetCounters="on")
template(name="logstash-index" type="list") {
constant(value="logstash-rsyslog-")
property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10")
}
template(name="plain-syslog" type="list") {
constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",") property(name="msg" position.from="2")
}
if ( $syslogfacility-text == 'syslog' ) then {
action( type="omelasticsearch"
template="plain-syslog"
server="10.13.57.35"
searchIndex="logstash-index"
searchType="impstats"
bulkmode="on"
dynSearchIndex="on"
)
stop
}
</code></pre>
</div>
<p>这里用到一个小窍门。impstats 只是 message 部分内容是 JSON 格式,那么如果合在总的内容里,可能就得跟老版的 logstash 事件格式一样,专门放在 <code class="highlighter-rouge">@fields:</code> 里面去了。但是,利用 <code class="highlighter-rouge">position.from</code> 参数,把 message 部分的开头 <code class="highlighter-rouge"><span class="p">{</span></code> 给删掉,就把整个内容都提升到顶层了,变成了新版 logstash 的事件格式了!</p>
<p>Rsyslog 的 template 语法多变,实现这个同样的目的,在 <code class="highlighter-rouge">mmjsonparse</code> 或者 <code class="highlighter-rouge">mmnormalize</code> 的配合下,就可以有不同写法:</p>
<ul>
<li>Rsyslog 官方的 <code class="highlighter-rouge">$!all-json</code> 玩法 <a href="http://www.rsyslog.com/json-elasticsearch/">Parsing JSON (CEE) Logs and Sending them to Elasticsearch</a></li>
<li>Rackspace 官博的 <code class="highlighter-rouge">subtree</code> 玩法 <a href="https://developer.rackspace.com/blog/rsyslog-and-elasticsearch/">rsyslog & ElasticSearch</a></li>
</ul>
<p>normalize 是 rsyslog 作者自己写的一个日志格式分析工具,搞怪的是它的文本格式示例文件后缀名叫 <code class="highlighter-rouge">.rb</code>,不是 Ruby,而是 RuleBase……</p>
<p>RuleBase 支持的<a href="http://www.liblognorm.com/files/manual/index.html">语法</a>不多:</p>
<ul>
<li>date-rfc3164: date as specified in rfc3164 (example: %date:date-rfc3164%)</li>
<li>date-rfc5424: date as specified in rfc5424 (example: %date:date-rfc5424%)</li>
<li>ipv4: IP adress (example: %ip:ipv4%)</li>
<li>number: sequence of numbers (example: %port:number%)</li>
<li>word: everything until the next blank (example: %host:word%)</li>
<li>char-to: the field will be defined by the sign in the additional information (example: %tag:char-to:\x3a%: (x3a means “:” in the additional information))</li>
<li>quoted-string: If a quoted string is present, a property can be filled with the whole string (example: %quote:quoted-string%)</li>
<li>date-iso: date in ISO format (example: %date:date-iso%)</li>
<li>time-24hr: detects time in 24hr format (example: %time:time-24hr%)</li>
<li>time-12hr: detects time in 12hr format (example: %time:time-12hr%)</li>
<li>iptables: parses IP tables messages and fills properties accordingly(example: %tables:iptables%)</li>
</ul>
<p>从这个格式设计也可以看出,主要还是用来分析系统日志比较多。</p>
<p>目前来看,使用 Rsyslog 做整套日志处理系统的话,在数据结构化这步,还是用 <a href="http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmexternal.html">mmexternal</a> 插件来完成比较合适。</p>
<p>mmexternal 模块类似 squid 的 <code class="highlighter-rouge">url_rewrite_program</code> ,都是支持用任意语言写的脚本,死循环接收 STDIN(可以配置传输 line 还是 json 格式),处理完成后(JSON 格式)输出给 STDOUT 即可。官方示例见:<a href="https://github.com/rsyslog/rsyslog/blob/master/plugins/external/messagemod/anon_cc_nbrs/anon_cc_nbrs.py">https://github.com/rsyslog/rsyslog/blob/master/plugins/external/messagemod/anon_cc_nbrs/anon_cc_nbrs.py</a>。性能如何,有待测试了。</p>
LogStash::Inputs::Syslog 性能测试与优化
2014-10-18T08:01:00+08:00
logstash
syslog
ruby
netty
http://chenlinux.com/2014/10/18/performance-testing-tunning-for-logstash-inputs-syslog
<p>最近因为项目需要,必须想办法提高 logstash indexer 接收 rsyslog 转发数据的性能。首先,就是要了解 logstash 到底能收多快?</p>
<p>之前用 libev 库写过类似功能的程序,所以一开始也是打算找个能在 JRuby 上运行的 netty 封装。找到了 <a href="https://github.com/m0wfo/foxbat">foxbat</a> 库,不过最后发现效果跟官方的标准 socket 实现差不多。(这部分另篇讲述)</p>
<p>后来又发现另一个库:<a href="https://github.com/jordansissel/experiments/tree/master/ruby/jruby-netty/syslog-server">jruby-netty</a>,注意到这个作者就是 logstash 作者 jordansissel!</p>
<p>当然,最终并不是用上这个项目的代码来改写 logstash,而是从这里面学到了如何方便的进行 syslog server 性能压测。测试方式:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>yes "<44>May 19 18:30:17 snack jls: foo bar 32" | nc localhost 3000
</code></pre>
</div>
<p>或者</p>
<div class="highlighter-rouge"><pre class="highlight"><code>loggen -r 500000 -iS -s 120 -I 50 localhost 3000
</code></pre>
</div>
<p>loggen 是 syslog-ng 带的工具,还得另外安装。而上面第一行的方式,这个 <code class="highlighter-rouge">yes</code> 用的真是绝妙!</p>
<p>就用这个测试方法,最终发现单机上 LogStash::Inputs::Syslog 的每秒处理能力只有 700 条:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>input {
syslog {
port => 3000
}
}
output {
stdout {
codec => dots
}
}
</code></pre>
</div>
<p>logstash 配置文件见上。然后测试启动命令如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>./bin/logstash -f syslog.conf | pv -abt > /dev/null
</code></pre>
</div>
<p><em>注意,centos 上的 pv 命令可能还没有 <code class="highlighter-rouge">-a</code> 参数。</em></p>
<p>为了逐一排除性能瓶颈。我依次注释掉了 <code class="highlighter-rouge">lib/logstash/inputs/syslog.rb</code> 中 <code class="highlighter-rouge">@date_filters.filter(event)</code> 和 <code class="highlighter-rouge">@grok_filters.filter(event)</code> 两段,并重新运行上次的测试。结果发现:</p>
<ul>
<li>TCPServer 接收的性能是每秒 50k 条</li>
<li>TCPServer 接收并完成 grok filter 的性能是每秒 5k 条</li>
<li>TCPServer 接收并完成 grok 和 date filter 的性能是每秒 700 条</li>
</ul>
<p>性能成几何级的下降!</p>
<p>而另外通过 <code class="highlighter-rouge">input { generator { count => 3000000 } }</code> 测试可以发现,logstash 本身空数据流转的性能也不过就是每秒钟几万条。所以,优化点就在后面的 filter 上。</p>
<p><em>注:空数据流转的测试采用 inputs/generator 插件</em></p>
<p>LogStash::Inputs::Syslog 中,TCPServer 对每个 client 单独开一个 Thread,但是这个 Thread 内要顺序完成 <code class="highlighter-rouge">@codec.decode</code>,<code class="highlighter-rouge">@grok_filter.filter</code> 和 <code class="highlighter-rouge">@date_filter.filter</code> 三大步骤后,才算完成。而我们都知道:Logstash 配置中 filter 阶段的插件是可以多线程完成的。所以,解决办法就来了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>input {
tcp {
port => 3000
}
}
filter {
grok {
overwrite => "message"
match => ["message", "<\d+>%{SYSLOGLINE}"]
}
date {
locale => "en"
match => ["timestamp", "MMM dd HH:mm:ss", "MMM d HH:mm:ss"]
}
}
output {
stdout {
codec => dots
}
}
</code></pre>
</div>
<p>然后重新测试,发现性能提高到了每秒 4.5k。再用下面命令运行测试:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> ./bin/logstash -f syslog.conf -w 20 | pv -bt > /dev/null
</code></pre>
</div>
<p>发现性能提高到了每秒 30 k 条!</p>
<p>此外,还陆续完成了另外一些测试。</p>
<p>比如:</p>
<ul>
<li>outputs/elasticsearch 的 protocol 使用 node 还是 http 的问题。测试在单台环境下,node 只有 5k 的 indexing 速度,而 http 有7k。</li>
<li>在 inputs/file 的前提下,outputs/stdout{dots} 比 outputs/elasticsearch{http} 处理速度快一倍,即有 15k。</li>
<li>下载了 heka 的二进制包,通过下面配置测试其接受 syslog 输入,并以 logstash 的 schema 输出到文件的性能。结果是每秒 30k,跟之前优化后的 logstash 基本一致。</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nn">[hekad]</span>
<span class="py">maxprocs</span> <span class="p">=</span> <span class="s">48</span>
<span class="nn">[TcpInput]</span>
<span class="py">address</span> <span class="p">=</span> <span class="s">":5140"</span>
<span class="py">parser_type</span> <span class="p">=</span> <span class="s">"token"</span>
<span class="py">decoder</span> <span class="p">=</span> <span class="s">"RsyslogDecoder"</span>
<span class="nn">[RsyslogDecoder]</span>
<span class="py">type</span> <span class="p">=</span> <span class="s">"SandboxDecoder"</span>
<span class="py">filename</span> <span class="p">=</span> <span class="s">"lua_decoders/rsyslog.lua"</span>
<span class="nn">[RsyslogDecoder.config]</span>
<span class="py">type</span> <span class="p">=</span> <span class="s">"mweibo"</span>
<span class="py">template</span> <span class="p">=</span> <span class="s">'<%pri%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n'</span>
<span class="py">tz</span> <span class="p">=</span> <span class="s">"Asia/Shanghai"</span>
<span class="nn">[ESLogstashV0Encoder]</span>
<span class="py">es_index_from_timestamp</span> <span class="p">=</span> <span class="s">true</span>
<span class="py">fields</span> <span class="p">=</span> <span class="s">["Timestamp", "Payload", "Hostname", "Fields"]</span>
<span class="py">type_name</span> <span class="p">=</span> <span class="s">"%{Type}"</span>
<span class="c"># [ElasticSearchOutput]
# message_matcher = "Type == 'nginx.access'"
# server = "http://10.13.57.35:9200"
# encoder = "ESLogstashV0Encoder"
# flush_interval = 50
# flush_count = 5000
</span>
<span class="nn">[counter_output]</span>
<span class="py">type</span> <span class="p">=</span> <span class="s">"FileOutput"</span>
<span class="py">path</span> <span class="p">=</span> <span class="s">"/tmp/debug.log"</span>
<span class="py">message_matcher</span> <span class="p">=</span> <span class="s">"TRUE"</span>
<span class="py">encoder</span> <span class="p">=</span> <span class="s">"ESLogstashV0Encoder"</span>
</code></pre>
</div>
<p>heka 文档称 maxprocs 设置为 cpu 数的两倍。不过实际测试中,不配置跟配置总共也就差一倍的性能。</p>
在 JRuby 上用 netty 模拟 eventmachine
2014-10-17T00:00:00+08:00
ruby
java
http://chenlinux.com/2014/10/17/netty-to-eventmachine-on-jruby
<p>上一篇说到在 JRuby 上利用 netty 库实现事件驱动。事实上,为了让 Ruby 程序员更习惯,foxbat 模块是把 netty 库封装成 eventmachine 的接口来提供给用户使用的。所以,我们可以把程序写得更通用一些:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">if</span> <span class="n">defined?</span><span class="p">(</span><span class="no">JRUBY_VERSION</span><span class="p">)</span>
<span class="nb">require</span> <span class="s1">'foxbat'</span>
<span class="k">end</span>
<span class="nb">require</span> <span class="s1">'eventmachine'</span>
<span class="nb">require</span> <span class="s1">'socket'</span>
<span class="k">module</span> <span class="nn">SyslogRecv</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">options</span><span class="p">)</span>
<span class="vi">@output_queue</span> <span class="o">=</span> <span class="n">options</span><span class="p">[</span><span class="ss">:queue</span><span class="p">]</span>
<span class="vi">@codec</span> <span class="o">=</span> <span class="n">options</span><span class="p">[</span><span class="ss">:codec</span><span class="p">]</span>
<span class="vi">@grok_filter</span> <span class="o">=</span> <span class="n">options</span><span class="p">[</span><span class="ss">:grok_filter</span><span class="p">]</span>
<span class="vi">@date_filter</span> <span class="o">=</span> <span class="n">options</span><span class="p">[</span><span class="ss">:date_filter</span><span class="p">]</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">syslog_relay</span><span class="p">(</span><span class="n">event</span><span class="p">)</span>
<span class="vi">@grok_filter</span><span class="p">.</span><span class="nf">filter</span><span class="p">(</span><span class="n">event</span><span class="p">)</span>
<span class="k">if</span> <span class="n">event</span><span class="p">[</span><span class="s2">"tags"</span><span class="p">].</span><span class="nf">nil?</span> <span class="o">||</span> <span class="o">!</span><span class="n">event</span><span class="p">[</span><span class="s2">"tags"</span><span class="p">].</span><span class="nf">include?</span><span class="p">(</span><span class="s2">"_grokparsefailure"</span><span class="p">)</span>
<span class="n">event</span><span class="p">[</span><span class="s2">"timestamp"</span><span class="p">]</span> <span class="o">=</span> <span class="n">event</span><span class="p">[</span><span class="s2">"timestamp8601"</span><span class="p">]</span> <span class="k">if</span> <span class="n">event</span><span class="p">.</span><span class="nf">include?</span><span class="p">(</span><span class="s2">"timestamp8601"</span><span class="p">)</span>
<span class="vi">@date_filter</span><span class="p">.</span><span class="nf">filter</span><span class="p">(</span><span class="n">event</span><span class="p">)</span>
<span class="k">else</span>
<span class="vi">@logger</span><span class="p">.</span><span class="nf">info?</span> <span class="o">&&</span> <span class="vi">@logger</span><span class="p">.</span><span class="nf">info</span><span class="p">(</span><span class="s2">"NOT SYSLOG"</span><span class="p">,</span> <span class="ss">:message</span> <span class="o">=></span> <span class="n">event</span><span class="p">[</span><span class="s2">"message"</span><span class="p">])</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">post_init</span>
<span class="p">(</span><span class="vc">@@connections</span> <span class="o">||=</span> <span class="p">[])</span> <span class="o"><<</span> <span class="nb">self</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">receive_data</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="vc">@@connections</span><span class="p">.</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">client</span><span class="o">|</span>
<span class="k">if</span> <span class="n">defined?</span><span class="p">(</span><span class="no">JRUBY_VERSION</span><span class="p">)</span>
<span class="n">ip</span> <span class="o">=</span> <span class="n">client</span><span class="p">.</span><span class="nf">get_peername</span><span class="p">.</span><span class="nf">getAddress</span><span class="p">.</span><span class="nf">getHostAddress</span>
<span class="n">port</span> <span class="o">=</span> <span class="n">client</span><span class="p">.</span><span class="nf">get_peername</span><span class="p">.</span><span class="nf">getPort</span>
<span class="k">else</span>
<span class="n">port</span><span class="p">,</span> <span class="n">ip</span> <span class="o">=</span> <span class="no">Socket</span><span class="p">.</span><span class="nf">unpack_sockaddr_in</span><span class="p">(</span><span class="n">client</span><span class="p">.</span><span class="nf">get_peername</span><span class="p">)</span>
<span class="k">end</span>
<span class="o">::</span><span class="no">LogStash</span><span class="o">::</span><span class="no">Util</span><span class="o">::</span><span class="n">set_thread_name</span><span class="p">(</span><span class="s2">"input|syslog|tcp|</span><span class="si">#{</span><span class="n">ip</span><span class="si">}</span><span class="s2">:</span><span class="si">#{</span><span class="n">port</span><span class="si">}</span><span class="s2">}"</span><span class="p">)</span>
<span class="vi">@codec</span><span class="p">.</span><span class="nf">decode</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">event</span><span class="o">|</span>
<span class="n">event</span><span class="p">[</span><span class="s2">"host"</span><span class="p">]</span> <span class="o">=</span> <span class="n">ip</span>
<span class="n">syslog_relay</span><span class="p">(</span><span class="n">event</span><span class="p">)</span>
<span class="vi">@output_queue</span> <span class="o"><<</span> <span class="n">event</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="n">output_queue</span><span class="p">)</span>
<span class="vi">@logger</span><span class="p">.</span><span class="nf">info</span><span class="p">(</span><span class="s2">"Starting syslog tcp listener"</span><span class="p">,</span> <span class="ss">:address</span> <span class="o">=></span> <span class="s2">"</span><span class="si">#{</span><span class="vi">@host</span><span class="si">}</span><span class="s2">:</span><span class="si">#{</span><span class="vi">@port</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="no">EventMachine</span><span class="o">::</span><span class="n">run</span> <span class="k">do</span>
<span class="no">EventMachine</span><span class="o">::</span><span class="n">start_server</span> <span class="vi">@host</span><span class="p">,</span> <span class="vi">@port</span><span class="p">,</span> <span class="no">SyslogRecv</span><span class="p">,</span> <span class="p">{</span>
<span class="ss">:queue</span> <span class="o">=></span> <span class="n">output_queue</span><span class="p">,</span>
<span class="ss">:codec</span> <span class="o">=></span> <span class="vi">@codec</span><span class="p">,</span>
<span class="ss">:grok_filter</span> <span class="o">=></span> <span class="vi">@grok_filter</span><span class="p">,</span>
<span class="ss">:date_filter</span> <span class="o">=></span> <span class="vi">@date_filter</span>
<span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p>初次用 EventMachine,发现写法还蛮奇怪的。<code class="highlighter-rouge">start_server</code> 传递参数必须是 module 或者 class,然后变量只能随后通过额外的哈希传递进去。</p>
<p>木有看 CPP 的 EM 实现,看这里 foxbat 的实现,发现在 JRuby 里使用 Java 还真是简单啊:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/env ruby</span>
<span class="nb">require</span> <span class="s2">"java"</span>
<span class="nb">require</span> <span class="no">File</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">dirname</span><span class="p">(</span><span class="kp">__FILE__</span><span class="p">),</span> <span class="s2">"netty-3.2.4.Final.jar"</span><span class="p">)</span>
<span class="nb">require</span> <span class="no">File</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">dirname</span><span class="p">(</span><span class="kp">__FILE__</span><span class="p">),</span> <span class="s2">"syslogdecoder.jar"</span><span class="p">)</span>
<span class="n">java_import</span> <span class="s2">"com.loggly.syslog.SyslogDecoder"</span>
<span class="n">java_import</span> <span class="s2">"org.jboss.netty.channel.SimpleChannelHandler"</span>
<span class="n">java_import</span> <span class="s2">"org.jboss.netty.channel.ChannelPipelineFactory"</span>
<span class="n">java_import</span> <span class="s2">"org.jboss.netty.channel.Channels"</span>
<span class="n">java_import</span> <span class="s2">"org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory"</span>
<span class="n">java_import</span> <span class="s2">"org.jboss.netty.bootstrap.ServerBootstrap"</span>
<span class="k">class</span> <span class="nc">SyslogServerHandler</span> <span class="o"><</span> <span class="no">SimpleChannelHandler</span>
<span class="k">class</span> <span class="o"><<</span> <span class="nb">self</span>
<span class="kp">include</span> <span class="no">ChannelPipelineFactory</span>
<span class="k">def</span> <span class="nf">getPipeline</span>
<span class="k">return</span> <span class="no">Channels</span><span class="p">.</span><span class="nf">pipeline</span><span class="p">(</span><span class="no">SyslogDecoder</span><span class="p">.</span><span class="nf">new</span><span class="p">,</span> <span class="nb">self</span><span class="p">.</span><span class="nf">new</span><span class="p">)</span>
<span class="k">end</span> <span class="c1"># def getPipeline</span>
<span class="k">end</span> <span class="c1"># class << self</span>
<span class="k">def</span> <span class="nf">initialize</span>
<span class="k">super</span>
<span class="k">end</span> <span class="c1"># def initialize</span>
<span class="k">def</span> <span class="nf">messageReceived</span><span class="p">(</span><span class="n">context</span><span class="p">,</span> <span class="n">event</span><span class="p">)</span>
<span class="n">e</span> <span class="o">=</span> <span class="n">event</span><span class="p">.</span><span class="nf">getMessage</span><span class="p">.</span><span class="nf">toString</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)</span>
<span class="k">end</span> <span class="c1"># def messageReceived</span>
<span class="k">def</span> <span class="nf">exceptionCaught</span><span class="p">(</span><span class="n">context</span><span class="p">,</span> <span class="n">exception</span><span class="p">)</span>
<span class="n">exception</span><span class="p">.</span><span class="nf">getCause</span><span class="p">.</span><span class="nf">printStackTrace</span>
<span class="n">exception</span><span class="p">.</span><span class="nf">getChannel</span><span class="p">.</span><span class="nf">close</span>
<span class="k">end</span> <span class="c1"># def exceptionCaught</span>
<span class="k">end</span> <span class="c1"># class SyslogServerHandler</span>
<span class="k">class</span> <span class="nc">RubySyslogServer</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">host</span><span class="p">,</span> <span class="n">port</span><span class="p">)</span>
<span class="vi">@factory</span> <span class="o">=</span> <span class="no">NioServerSocketChannelFactory</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span>
<span class="n">java</span><span class="p">.</span><span class="nf">util</span><span class="p">.</span><span class="nf">concurrent</span><span class="o">.</span><span class="no">Executors</span><span class="p">.</span><span class="nf">newCachedThreadPool</span><span class="p">(),</span>
<span class="n">java</span><span class="p">.</span><span class="nf">util</span><span class="p">.</span><span class="nf">concurrent</span><span class="o">.</span><span class="no">Executors</span><span class="p">.</span><span class="nf">newCachedThreadPool</span><span class="p">()</span>
<span class="p">)</span>
<span class="vi">@bootstrap</span> <span class="o">=</span> <span class="no">ServerBootstrap</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="vi">@factory</span><span class="p">)</span>
<span class="vi">@bootstrap</span><span class="p">.</span><span class="nf">setPipelineFactory</span><span class="p">(</span><span class="no">SyslogServerHandler</span><span class="p">)</span>
<span class="vi">@bootstrap</span><span class="p">.</span><span class="nf">setOption</span><span class="p">(</span><span class="s2">"child.tcpNoDelay"</span><span class="p">,</span> <span class="kp">true</span><span class="p">);</span>
<span class="vi">@bootstrap</span><span class="p">.</span><span class="nf">setOption</span><span class="p">(</span><span class="s2">"child.keepAlive"</span><span class="p">,</span> <span class="kp">true</span><span class="p">);</span>
<span class="vi">@host</span> <span class="o">=</span> <span class="n">host</span>
<span class="vi">@port</span> <span class="o">=</span> <span class="n">port</span>
<span class="k">end</span> <span class="c1"># def initialize</span>
<span class="k">def</span> <span class="nf">start</span>
<span class="n">address</span> <span class="o">=</span> <span class="n">java</span><span class="p">.</span><span class="nf">net</span><span class="o">.</span><span class="no">InetSocketAddress</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="vi">@host</span><span class="p">,</span> <span class="vi">@port</span><span class="p">)</span>
<span class="k">return</span> <span class="vi">@bootstrap</span><span class="p">.</span><span class="nf">bind</span><span class="p">(</span><span class="n">address</span><span class="p">)</span>
<span class="k">end</span> <span class="c1"># def start</span>
<span class="k">end</span> <span class="c1"># class SyslogServer</span>
<span class="k">if</span> <span class="kp">__FILE__</span> <span class="o">==</span> <span class="vg">$0</span>
<span class="n">host</span> <span class="o">=</span> <span class="no">ARGV</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">port</span> <span class="o">=</span> <span class="no">ARGV</span><span class="p">[</span><span class="mi">1</span><span class="p">].</span><span class="nf">to_i</span>
<span class="no">RubySyslogServer</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">host</span><span class="p">,</span> <span class="n">port</span><span class="p">).</span><span class="nf">start</span>
<span class="k">end</span>
</code></pre>
</div>
<p>直接加载 jar 包,导入各种类。然后就能照样用了。</p>
PerlAPI 里的 Magic 简介
2014-10-11T00:00:00+08:00
perl
http://chenlinux.com/2014/10/11/perlapi-magic-intro
<p>前几天看到 cindylinz 发了一个新 CPAN 模块叫 <a href="https://metacpan.org/pod/Scalar::Watcher">Scalar::Watcher</a>,有朋友问我这个是怎么实现的,在无限循环啊,多线程啊,IO 阻塞啊等情况下,还能被触发么?</p>
<p>于是我去仔细看了一下这个模块的代码。最关键的就是下面这几行:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">SvUPGRADE</span><span class="p">(</span><span class="n">target</span><span class="p">,</span> <span class="n">SVt_PVMG</span><span class="p">);</span>
<span class="n">sv_magicext</span><span class="p">(</span><span class="n">target</span><span class="p">,</span> <span class="n">handler_cv</span><span class="p">,</span> <span class="n">PERL_MAGIC_ext</span><span class="p">,</span> <span class="o">&</span><span class="n">modified_vtbl</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
</code></pre>
</div>
<p>这里其实用的是 perlapi 里的 Magic:</p>
<ul>
<li>
<p>第一行,设置监听变量为 <code class="highlighter-rouge">SVt_PVMG</code>,即带有 Magic 的标量;</p>
<p><code class="highlighter-rouge">SvUPGRADE</code> 函数见 perlguts 文档的 “<a href="http://perldoc.perl.org/perlguts.html#Assigning-Magic">Assigning Magic</a>” 部分。</p>
</li>
<li>
<p>第二行,设置该变量的 Magic 扩展,即往标量的 Magic 链表上加内容。</p>
<p><code class="highlighter-rouge">sv_magicext</code> 函数说明见 perlapi 文档的 “<a href="http://perldoc.perl.org/perlapi.html#SV-Body-Allocation">SV-Body Allocation</a>” 部分。</p>
</li>
</ul>
<p>Magic 主要有两个作用,一个叫 Hook Method,一个叫 Managed Data。我们都很熟悉的 Moose 框架就是利用的 Magic 的 Managed Data 实现的。而这里,用到的是 Hook Method。</p>
<p>Scalar::Watcher 模块文档较少,虽然好用但是不好懂。我在 CPAN 上发现另一个模块,<a href="https://metacpan.org/pod/Variable::Magic">Variable::Magic</a> 。文档写的很详细。其中的 <code class="highlighter-rouge">set</code> 方法就是跟 Scalar::Watcher 类似的作用,大家可以读一读这个模块的文档。</p>
<p>所以可以回答朋友的问题了,在循环之类的地方每次都可以触发没问题。但是如果你在回调函数里面做阻塞操作,那肯定也是堵塞的。</p>
从源代码运行 Kibana 4
2014-10-10T00:00:00+08:00
logstash
kibana
ruby
nodejs
http://chenlinux.com/2014/10/10/run-kibana4-without-jar
<p>Kibana 4 发布了,出人意料的是提供的居然是一个 jar 包的运行方式。好在有源码可看,根据源码可以分析得知,v4 版其实是一个 angularjs 写的 kibana 配上一个 sinatra 写的 proxyserver。这么一来,我们也就知道怎么来从源代码运行 Kibana 4,而不是用 Java 启动了。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># 安装 nodejs 和 npm 命令,仅用于下载依赖包,实际运行不需要</span>
port install nodejs npm
<span class="c"># 下载 kibana 4 源码</span>
git clone https://github.com/elasticsearch/kibana.git kibana4
<span class="nb">cd </span>kibana4/
<span class="c"># 安装 bower 工具</span>
npm install -g bower
<span class="c"># 读取目录中的 bower.json,</span>
<span class="c"># 依此下载所有 js 依赖库到其中定义的路径</span>
<span class="c"># src/kibana/bower_components 下</span>
bower install
<span class="nb">cd </span>src/server
<span class="c"># 安装 bundler 工具</span>
gem install bundler
<span class="c"># 读取目录中的 Gemfile,</span>
<span class="c"># 依此安装所有的 RubyGem 依赖库</span>
bundle install
<span class="c"># 安装 lessc 工具</span>
npm install -g less
<span class="c"># kibana 4 源码中在导入 lesshat 的时候都没写具体路径,所以要切换到对应目录下执行</span>
<span class="nb">cd</span> ../src/kibana/bower_components/lesshat/build
<span class="c"># 编译 kibana 内的 *.less 文件为 *.css 文件</span>
<span class="k">for </span>i <span class="k">in</span> <span class="sb">`</span>find ../../.. -name <span class="s1">'[a-z]*.less'</span> | grep -v bower_components<span class="sb">`</span>;<span class="k">do</span>
../../../../../node_modules/.bin/lessc <span class="nv">$i</span> <span class="k">${</span><span class="nv">i</span><span class="p">/.less/.css/</span><span class="k">}</span>
<span class="k">done</span>
<span class="c"># 进入代理服务器目录</span>
<span class="nb">cd</span> ../../../../server/
<span class="c"># 启动 sinatra 服务器</span>
./bin/initialize
</code></pre>
</div>
<p>这样就可以通过 “localhost:5601” 访问了。</p>
<p>此外,Elasticsearch 集群的地址,在 <code class="highlighter-rouge">src/server/config/kibana.yml</code> 中配置。注意里面的 <code class="highlighter-rouge">kibana-int</code> 建议大家使用的时候改个名儿,不然万一跟你原先 kibana3 的混合在一起了就不好了。</p>
<p>最后,如果你的集群版本低于 1.4.0.BETA1,也不要着急,其实目前代码并没有用上什么这个版本的特性,所以可以通过修改 <code class="highlighter-rouge">src/kibana/index.js</code> 改变这个版本检测:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="gd">--- a/src/kibana/index.js
</span><span class="gi">+++ b/src/kibana/index.js
</span><span class="gu">@@ -33,7 +33,7 @@ define(function (require) {
</span> // Use this for cache busting partials
.constant('cacheBust', window.KIBANA_COMMIT_SHA)
// The minimum Elasticsearch version required to run Kibana
<span class="gd">- .constant('minimumElasticsearchVersion', '1.4.0.Beta1')
</span><span class="gi">+ .constant('minimumElasticsearchVersion', '1.1.0')
</span> // When we need to identify the current session of the app, ef shard preference
.constant('sessionId', Date.now())
// attach the route manager's known routes
</code></pre>
</div>
<p>Kibana 4 的界面,改成了 Query -> Visual -> Dashboard 三个解耦层次。而且不再是固定的提供某种某种 panel,改成自己选择、拼接甚至书写 Aggr 聚合函数的方式来灵活的生成图表。可以说,对使用者的 ES 知识,要求更高了。</p>
<p>后续如何发展,大家一起关注吧。</p>
【翻译】Elasticsearch 1.4.0 beta 1 发版日志
2014-10-10T00:00:00+08:00
logstash
elasticsearch
http://chenlinux.com/2014/10/10/elasticsearch-1-4-beta-1-released
<p>原文见:<a href="http://www.elasticsearch.org/blog/elasticsearch-1-4-0-beta-released/">http://www.elasticsearch.org/blog/elasticsearch-1-4-0-beta-released/</a></p>
<hr />
<p>今天,我们很高兴公告基于 <strong>Lucene 4.10.1</strong> 的 <strong>Elasticsearch 1.4.0.Beta1</strong> 发布。你可以从这里下载并阅读完整的变更列表:<a href="http://www.elasticsearch.org/downloads/1-4-0-Beta1">Elasticsearch 1.4.0.Beta1</a>。</p>
<p>1.4.0 版的主题就是<strong>弹性</strong>:让 Elasticsearch 比过去更稳定更可靠。当所有东西都按照它应该的样子运行的时候,就很容易变得可靠了。但是不在意料中的事情发生时,复杂的部分就来了:节点内存溢出,它们的性能被慢垃圾回收或者超重的 I/O 拖累,网络连接失败,或者数据传输不规律。</p>
<p>这次 beta 版主要在三方面力图改善弹性:</p>
<ul>
<li>通过减少<a href="#section">内存使用</a>提供更好的节点稳定性。</li>
<li>通过改进发现算法提供更好的<a href="#section-1">集群稳定性</a>。</li>
<li>通过<a href="#checksums">checksums</a>提供更好的数据损坏检测。</li>
</ul>
<p>分布式系统是复杂的。我们已经有一个广泛的测试套件,可以创建随机场景,模拟我们自己都没想过的条件。但是依然会有无限多在此范围之外的情况。1.4.0.Beta1 里已经包含了我们目前能做到的各种优化努力。真心期望大家在实际运用中测试这些变更,然后<a href="https://github.com/elasticsearch/elasticsearch/issues">告诉我们你碰到的问题</a>。</p>
<h2 id="section">内存管理</h2>
<ul>
<li>内存压力</li>
<li>swap (参见 <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.4/setup-configuration.html#setup-configuration-memory">memory settings</a>)</li>
<li>太大的 heaps</li>
</ul>
<p>这次发版包括了一系列变更来提升内存管理,并由此提升节点稳定性:</p>
<h3 id="doc-values">doc values</h3>
<p><em>fielddata</em> 是最主要的内存大户。为了让聚合、排序以及脚本访问字段值时更快速,我们会加载字段值到内存,并保留在内存中。内存的堆空间非常宝贵,所以内存里的数据需要使用复杂的压缩算法和微优化来完成每次计算。正常情况下这样会工作的很好,直到你的数据大小超过了堆空间大小。这个问题看起来可以通过添加更多节点的方式解决。不过通常来说,堆空间问题总是会在 CPU 和 I/O 之前先到达瓶颈。</p>
<p>现有版本已经添加了 doc values 支持。本质上,doc values 提供了和内存中 fielddata 一样的功能,不过他们在写入索引的时候就直接落到了磁盘上。而好处就是:他们<strong>消耗很少的堆空间</strong>。Doc values 在读取的时候也不是从内存,而是从磁盘上读取。虽然访问磁盘很慢,但是 doc values 可以利用内核的文件系统缓存。文件系统缓存可不像 JVM 的堆,不会有 32GB 的限制。所以把 fielddata 从堆转移到文件系统缓存里,你只用消耗更小的堆空间,也意味着更快的垃圾回收,以及<strong>更稳定的节点</strong>。</p>
<p>在本次发版之前,doc values 明显慢于在内存里的 fielddata 。而这次我们显著提升了性能,几乎达到了和在内存里一样快的效果。</p>
<p>用doc values 替换内存 fielddata,你只需要向下面这样构建新字段就行:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="err">PUT</span><span class="w"> </span><span class="err">/my_index</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nt">"mappings"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"my_type"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"timestamp"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"date"</span><span class="p">,</span><span class="w">
</span><span class="nt">"doc_values"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
</div>
<p>有了这个映射表,要用这个字段数据都会自动从磁盘加载 doc values 而不是进到内存里。<strong>注意</strong>:目前 doc values 还不能在经过分词器的 <code class="highlighter-rouge">string</code> 字段上使用。</p>
<h3 id="request-circuit-breaker">request circuit breaker</h3>
<p>fielddata 断路器之前已经被加入,用作限制 fielddata 可用的最大内存,这是导致 OOM 的最大恶因。而限制,我们把这个机制扩展到<a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.4/index-modules-fielddata.html#request-circuit-breaker">请求界别</a>,用来限制每次请求可用的最大内存。</p>
<h3 id="bloom-filters">bloom filters</h3>
<p><a href="http://en.wikipedia.org/wiki/Bloom_filter">Bloom filters</a> 在写入索引时提供了重要的性能优化 – 用以检查是否有已存在的文档 id ,在通过 id 访问文档时,用来探测哪个 segment 包含这个文档。不过当然的,这也有代价,就是内存消耗。目前的改进是移除了对 bloom filters 的依赖。目前 Elasticsearch 只在写入索引(仅是真实用例上的经验,没有我们的测试用例证明)的时候构建它,但<a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.4/indices-update-settings.html#codec-bloom-load">默认</a>不再加载进内存。如果一切顺利的话,未来的版本里我们会彻底移除它。</p>
<h2 id="section-1">集群稳定性</h2>
<p>提高集群稳定性最大的工作就是提高节点稳定性。如果节点稳定且响应及时,就极大的减少了集群不稳定的可能。换句话说,我们活在一个不完美的世界 – 事情总是往意料之外发展,而集群就需要能无损的从这些情况中恢复回来。</p>
<p>我们在 improve_zen 分支上花了几个月的时间来提高 Elasticsearch 从失败中恢复的能力。首先,我们添加测试用例来复原复杂的网络故障。然后为每个测试用例添加补丁。肯定还有很多需要做的,不过目前来说,用户们已经碰到过的绝大多数问题我们已经解决了,包括<a href="https://github.com/elasticsearch/elasticsearch/issues/2488">issue #2488</a> – “minimum_master_nodes 在交叉脑裂时不起作用”。</p>
<p>我们非常认真的对待集群的弹性问题。希望你能明白 Elasticsearch 能为你做什么,也能明白它的弱点在哪。考虑到这点,我们创建了<a href="http://www.elasticsearch.org/guide/en/elasticsearch/resiliency/current/index.html">弹性状态文档</a>。这个文档记录了我们以及我们的用户碰到过各种弹性方面的问题,有些可能已经修复,有些可能还没有。请认真阅读这篇文档,采取适当的措施来保护你的数据。</p>
<h2 id="section-2">数据损坏探测</h2>
<p>从网络恢复过来的分片的 checksum 帮助我们发现过一个压缩库的 bug,这是 1.3.2 版本的时候发生的事情。从那天起,我们给 Elasticsearch 添加了越来越多的 checksum 认证。</p>
<ul>
<li>在合并时,segment 中的所有文件都有自己的 checksum 验证(<a href="https://github.com/elasticsearch/elasticsearch/issues/7360">#7360</a>).</li>
<li>重新开所有索引的时候,segment 里的小文件完整的验证,大文件则做轻量级的分段验证(<a href="https://issues.apache.org/jira/browse/LUCENE-5842">LUCENE-5842</a>).</li>
<li>从 transaction 日志重放事件的时候,每个事件都有自己的 checksum 验证(<a href="https://github.com/elasticsearch/elasticsearch/issues/6554">#6554</a>).</li>
<li>During shard recovery, or when restoring from a snapshot, Elasticsearch needs to compare a local file with a remote copy to ensure that they are identical. Using just the file length and checksum proved to be insufficient. Instead, we now check the identity of all the files in the segment (<a href="https://github.com/elasticsearch/elasticsearch/issues/7159">#7159</a>).</li>
</ul>
<h2 id="section-3">其他亮点</h2>
<p>你可以在 <a href="http://www.elasticsearch.org/downloads/1-4-0-Beta1">Elasticsearch 1.4.0.Beta1 changelog</a> 里读到这个版本的所有特性,功能和修复。不过还是有些小改动值得单独提一下的:</p>
<h3 id="groovy--mvel">groovy 代替了 mvel</h3>
<p>Groovy 现在成为了新的默认脚本语言。之前的 MVEL 太老了,而且它不能运行在沙箱里也带来了安全隐患。Groovy 是沙箱化的(这意味着可以放心的开启)(译者注:还记得1.2版本时候的所谓安全漏洞吧),而且 Groovy 有个很好的管理团队,运行速度也<strong>很快</strong>!更多信息见<a href="http://www.elasticsearch.org/blog/scripting/">博客关于脚本的内容</a>。</p>
<h3 id="cors">默认关闭 cors</h3>
<p>默认配置下的 Elasticsearch 很容易遭受跨站攻击。所以我们默认关闭掉 <a href="http://en.wikipedia.org/wiki/Cross-origin_resource_sharing">CORS</a>。Elasticsearch 里的 site 插件会照常工作,但是外部站点不再被允许访问远程集群,除非你再次打开 CORS。我们还添加了更多的<a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.4/modules-http.html#_settings_2">CORS 配置项</a>让你可以控制哪些站点可以被允许访问。更多信息请看我们的<a href="http://www.elasticsearch.org/community/security">安全页</a>。</p>
<h3 id="query-cache">请求缓存(query cache)</h3>
<p>一个新的实验性<a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.4/index-modules-shard-query-cache.html">分片层次的请求缓存</a>可以让在静态索引上的聚合请求瞬间返回响应。想想你有一个仪表板展示你的网站每天的 PV 数。这个书在过去的索引上不可能再变化了,但是聚合请求在每次页面刷新的时候都需要重新计算。有了新的请求缓存,聚合结果就可以直接从缓存中返回,除非分片中的数据发生了变化。你不用担心会从缓存中得到过期的结果 – 它永远都会跟没缓存一样。</p>
<h3 id="section-4">新的聚合函数</h3>
<p>我们添加了三个新的聚合函数:</p>
<p><code class="highlighter-rouge">filters</code></p>
<div class="highlighter-rouge"><pre class="highlight"><code>这是 `filter` 聚合的扩展。允许你定义多个桶(bucket),每个桶里有不同的过滤器。
</code></pre>
</div>
<p><code class="highlighter-rouge">children</code></p>
<div class="highlighter-rouge"><pre class="highlight"><code>相当于 `nested` 的父子聚合,`children` 可以针对属于某个父文档的子文档做聚合。
</code></pre>
</div>
<p><code class="highlighter-rouge">scripted_metric</code></p>
<div class="highlighter-rouge"><pre class="highlight"><code>给你完全掌控数据数值运算的能力。提供了在初始化、文档收集、分片层次合并,以及全局归并阶段的钩子。
</code></pre>
</div>
<h3 id="index-">获取 /index 的接口</h3>
<p>之前,你可以分别为一个索引获取他的别名,映射表,配置等等。而<a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.4/indices-get-index.html"><code class="highlighter-rouge">get-index</code> 接口</a> 现在让你可以一次获取一个或者多个索引的全部信息。这在你需要创建一个跟已有索引很类似或者几乎一样的新索引的时候,相当有用。</p>
<h3 id="section-5">索引写入和更新</h3>
<p>在文档写入和更新方面也有一些改进:</p>
<ul>
<li>我们现在用 <a href="http://boundary.com/blog/2012/01/12/flake-a-decentralized-k-ordered-unique-id-generator-in-erlang">Flake IDs</a> 自动生成文档的 ID。在查找主键的时候,能提供更好的性能。</li>
<li>如果设置 <code class="highlighter-rouge">detect_noop</code> 为 <code class="highlighter-rouge">true</code>,一个不做任何实际变动的更新操作现在消耗更小了。打开这个参数,就只有变更了 <code class="highlighter-rouge">_source</code> 字段内容的更新请求才能写入新版本文档。</li>
<li>更新操作可以完全由脚本控制。之前,脚本只能在字段已经存在的时候运行,否则会插入一个 <code class="highlighter-rouge">upsert</code> 文档。现在 <code class="highlighter-rouge">scripted_upsert</code> 参数允许你在脚本中直接处理文档创建工作。</li>
</ul>
<h3 id="function-score">function score</h3>
<p>非常有用的 <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.4/query-dsl-function-score-query.html"><code class="highlighter-rouge">function_score</code> 请求</a>现在支持权重参数,用来优化每个指定函数的相关性影响。这样你可以把更多权重给新近的而不是热点的,给价格而不是位置。此外,<code class="highlighter-rouge">random_score</code>函数不再被 segment 合并影响,增强了排序一致性。</p>
<h2 id="section-6">试一试</h2>
<p>请<a href="http://www.elasticsearch.org/downloads/1-4-0-Beta1">下载 Elasticsearch 1.4.0.Beta1</a>,尝试一下,然后在 Twitter 上<a href="https://twitter.com/elasticsearch">@elasticsearch</a>) 说出你的想法。你也可以在 <a href="https://github.com/elasticsearch/elasticsearch/issues">GitHub issues 页</a>上报告问题。</p>
【翻译】Kibana 4 beta 1 发版日志
2014-10-07T00:00:00+08:00
logstash
kibana
http://chenlinux.com/2014/10/07/kibana-4-beta-1-released
<p>原文地址见:<a href="http://www.elasticsearch.org/blog/kibana-4-beta-1-released/">http://www.elasticsearch.org/blog/kibana-4-beta-1-released/</a></p>
<hr />
<p>今天,我们<del>自豪高兴满意控制不住地兴奋过头欣喜若狂</del>相当高兴得给大家分享一下 Kibana 项目的未来,以及 Kibana 4 的第一个 beta 版本。</p>
<h2 id="section">我现在就要!快给我!</h2>
<p>从<a href="http://www.elasticsearch.org/overview/kibana/installation/">这里</a>下载,然后看 <a href="https://github.com/elasticsearch/kibana/blob/master/README.md">README.md</a> 里新的而且更简单的安装流程。当然,你最好还是读一下本文剩下的内容,有很多超棒的秘诀呢!</p>
<h2 id="kibana-4">欢迎来到 kibana 4</h2>
<p>我们正走在 Kibana 4 的漫漫长路上:可以预见还会有好几个 beta 版本,每个都有新的特性,可视化和改善。我们梳理了各种反馈、邮件列表、IRC 以及 Github 的 issue ,把特性加入到这个 beta1 版本中,真是罪孽深重。我们已经在为 beta2 版本努力工作,在此,很高兴分享一下我们的 roadmap,查看 Github 上打有 “<a href="https://github.com/elasticsearch/kibana/issues?q=is%3Aopen+is%3Aissue+label%3Aroadmap">Roadmap</a>” 标签的 issue。你们的反馈是我们永远做正确的事的保证。</p>
<p>反馈之外,我们回头想了想人们是怎么看数据的,更进一步,人们是怎么解决真实问题的。我们发现一个问题总是能引出另一些问题,而这些问题又能引出更多其他问题。如果你参加了 Monitotama,或者其他 Elasticsearch 见面会,你可能已经看到过 Kibana 4 概念性的原型演示。它可以让你创建更复杂的图标,Kibana 4 从 PoC 出发,扩展出一大堆新特性,让你编写问题,得到解答,然后解决之前从来没这么解决过的问题。</p>
<p>这种组合方式在 Kibana 4 中体现为聚合、搜索、可视化和仪表板融合在一起的方式。为了简化组成,我们把 Kibana 4 分成 3 个不同的界面,虽然一起工作,但是每个负责解决不同的一部分问题。</p>
<h2 id="section-1">熟悉的界面</h2>
<p>如果你是 Kibana 老用户,你会发现主页上 Discover 标签页的样子很熟悉。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/10/Screen-Shot-2014-09-30-at-4.07.15-PM.png" alt="" /></p>
<p>Discover 功能跟原先的带有一个文档表格和事件时间轴的搜索界面很像。在搜索框里输入,敲回车,然后让 Kibana 去挖掘你的 Elasticsearch 索引。说到索引,有一个快速下拉菜单让你在搜索的时候灵活的在多个索引之间切换。要切换回上一个索引,点击浏览器的回退按钮即可。不喜欢新的搜索关键词?同样点击回退按钮就能返回原来的搜索词了。当然,搜索框的历史中也存着过去的记录。</p>
<p>说道搜索,你既可以输入 Lucene Query String 语法,也可以用上一个经常被要求的特性,<strong>Elasticsearch JSON 搜索</strong> 到搜索框里。我们知道 JSON 格式可能比较难输对,所以不管你输入的是 Lucene Query String 还是 JSON,我们都会在发送给 Elasticsearch 之前替你验证一遍语法。不管你在 Kibana 4 的任何位置输入请求,这点都是生效的。</p>
<p>这样搜索也可以保存下来留待后用。重要的是:搜索不在绑定在仪表板上,他们可以在 Discover 页上再次调用,也可以运用在可能稍后才添加到仪表板上的可视化页里。因为,不管你在仪表板的哪一屏,<strong>搜索一直都会通过 URL 传递</strong>,所以链接到搜索非常简单。</p>
<h2 id="section-2">画图的在这里</h2>
<p>Kibana 4 的 Visualize 标签是之前说的概念原型里最高潮的地方。Kibana 4 把 Elasticsearch 的 nested 聚合函数的威力带到鼠标点击上。比如我想知道哪些国家访问我的网站,什么时候访问的,他们是否登录认证了?通过一个 canvas 上的单一请求,我就可以问出上面这些问题,然后看到结果是怎么相互联系的:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/10/Screen-Shot-2014-10-01-at-2.28.29-PM.png" alt="" /></p>
<p>Kibana 3 的时候,时间只能在 histogram 面板上显示,而 terms 只能在柱状图上显示。Kibana 4 可以利用多个 <strong>Elasticsearch 聚合函数</strong>。这包括 bucket 和 metric 聚合函数,其中有备受期待的<strong>基数</strong>(又叫唯一计数)聚合函数,更多支持还在实现中。我们不得不创建了一个全新的可视化框架来处理复杂的聚合函数。目前有三种支持的类型:柱状图,线状图和光圈图。同样,更多支持还在实现中。未来每个 Kibana 4 的 beta 版本都值得你期待。</p>
<p>光圈图类似多层次的饼图。理论上它可以有无限的环:</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/10/Screen-Shot-2014-10-01-at-12.49.50-PM.png" alt="" /></p>
<p>柱状图现在还不单单可以做时间。这里我们展示根据文件后缀名分解文件大小范围。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/10/Screen-Shot-2014-10-01-at-1.03.03-PM.png" alt="" /></p>
<p>现在你可能已经注意到每个可视化页底部的灰色小条。点击它,就可以看到图背后的源数据,然后,在大众要求下,提供了<strong>导出到CSV</strong> 以便后续分析的功能。你还可以看到 Elasticsearch 请求和响应的内容,以及请求的处理耗时。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/10/Screen-Shot-2014-10-01-at-12.31.14-PM.png" alt="" /></p>
<p>Visualization 既可以互动式搜索创建,让你在建图的时候修改请求,也可以关联到一个之前通过 Discover 标签创建保存的请求上。这样你可以关联一个请求到多个可视化页,如果需要更新一个搜索参数,只需要更新单独一个请求就行了。比如,假设你有多个图表,是用下面语句搜索图片内容的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>png OR jpg
</code></pre>
</div>
<p>保存成 “Images”。然后你打算支持动态 GIF 格式,你只需要更新 “Images” 的内容然后保存即可。所有关联了 “Images” 请求的图都会自动应用变更。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/10/Screen-Shot-2014-10-01-at-1.17.08-PM.png" alt="" />]</p>
<h2 id="section-3">给我看更多的图!</h2>
<p>当然,你依然可以创建令人惊叹的仪表板,而且它们现在更方便创建和管理了。过去那堆凌乱的配置框一去不复返了。添加进仪表板的每个面板都可以在 Visualize 标签页理创建、保存,并且重复利用。就像保存了的搜索可以在多个 visualizations 里使用一样,保存了的 visualization 也可以在多个仪表板里使用。你需要更新一个 visualization 的话,只需要在一个地方修改好,每个仪表板里的都会应用变更。</p>
<p>更进一步,虽然请求和可视化是绑定到一个选定的索引的,仪表板却不用。<strong>一个仪表板可以有从不同索引来的可视化</strong>。这意味着,你可以从你的用户索引关联到网站流量索引,从销售数据关联到市场研究再关联到气象站日志。这些都可以在同一屏上!</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/10/Screen-Shot-2014-10-01-at-1.19.39-PM.png" alt="" /></p>
<h2 id="section-4">更多</h2>
<p>一篇博客里完全不够说完全部内容,所以去下载安装然后亲自试试 <a href="http://www.elasticsearch.org/overview/kibana/installation/">HERE</a> 吧。如果你来自 Kibana 3,我们收集了一个小小的 FAQ 解释:<a href="https://github.com/elasticsearch/kibana/blob/master/K3_FAQ.md">HERE</a>。还是老话,我们需要你的反馈,构建 Kibana 4 的每一天,我们都用得着这些反馈,而我们也会继续让 Kibana 变得更好,更快,更简单。</p>
【翻译】Kibana 3 升级到 4 的常见问答
2014-10-07T00:00:00+08:00
logstash
kibana
http://chenlinux.com/2014/10/07/kibana-3-migration-faq
<p>原文见<a href="https://github.com/elasticsearch/kibana/blob/master/K3_FAQ.md">https://github.com/elasticsearch/kibana/blob/master/K3_FAQ.md</a>。</p>
<p>问:我在 Kibana 3 里最想要的某某特性有了么?<br />
答:就会有了!我们已经以 ticket 形式发布了目前的 roadmap。查看 GitHub 上的 beta 里程碑,看看有没有你想要的特性。</p>
<p>问:仪表板模式是否兼容?<br />
答:不好意思,不兼容了。要创建我们想要的新特性,还是用原先的模式是不可能的。Aggregation 跟 Facet 请求从根本上工作方式就不一样,新的仪表板不再绑定成行和列的样式,而且搜索框,可视化和仪表板的关系过于复杂,我们不得不重新设计一遍,来保证它的灵活可用。</p>
<p>问:怎么做多项搜索?<br />
答:”filters” Aggregation 可以运行你输入多项搜索条件然后完成可视化。甚至你可以在这里面自己写 JSON。</p>
<p>问:模板化/脚本化仪表板还在么?<br />
答:看看 URL 吧。每个应用的状态都记录在那里面,包括所有的过滤器,搜索和列。现在构建脚本化仪表板比过去简单多了。URL 是采用 RISON 编码的。</p>
<h3 id="section">译者注:</h3>
<p>RISON 是一个跟 JSON 很类似,还节省不少长度的东西。其官网见:<a href="http://mjtemplate.org/examples/rison.html">http://mjtemplate.org/examples/rison.html</a>。但是我访问看似乎已经挂了,更多一点的说明可以看<a href="https://github.com/Nanonid/rison">https://github.com/Nanonid/rison</a>。</p>
Mojolicious 应用的自定义子命令
2014-10-01T00:00:00+08:00
perl
mojolicious
http://chenlinux.com/2014/10/01/custom-mojolicious-app-command
<p>Mojolicious 框架开发应用的时候,可以跟 RoR 一样通过一系列子命令简化很多复杂操作。最简单的来说,就是快速生成整个 web 项目目录:<code class="highlighter-rouge">mojo generate youapp</code>。更多子命令见:<a href="http://cpan.php-oa.com/perldoc/Mojolicious/Commands">http://cpan.php-oa.com/perldoc/Mojolicious/Commands</a></p>
<p>其实我们还可以自己扩展这个子命令方式,实现自己的子命令。如果打算继续使用 <code class="highlighter-rouge">mojo subcommand</code> 的方式,那就把自己的子命令模块叫做 <code class="highlighter-rouge">Mojolicious::Command::yourcommand</code>,而如果打算在自己的名字空间下使用,比如叫 <code class="highlighter-rouge">MyApp::Command::mycommand</code>,那么需要在 <code class="highlighter-rouge">MyApp.pm</code> 里加一行代码,设置一下名字空间:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">sub </span><span class="nf">startup</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$self</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="nb">push</span> <span class="nv">@</span><span class="p">{</span><span class="nv">$self</span><span class="o">-></span><span class="nv">commands</span><span class="o">-></span><span class="nv">namespaces</span><span class="p">},</span> <span class="s">'MyApp::Command'</span><span class="p">;</span>
<span class="o">...</span>
<span class="p">};</span>
</code></pre>
</div>
<p>然后就可以写自己的 <code class="highlighter-rouge">MyApp::Command::mycommand</code> 了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nb">package</span> <span class="nn">MyApp::Command::</span><span class="nv">mycommand</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Mojo::</span><span class="nv">Base</span> <span class="s">'Mojolicious::Command'</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Mojo::</span><span class="nv">UserAgent</span><span class="p">;</span>
<span class="nv">has</span> <span class="nv">usage</span> <span class="o">=></span> <span class="s">"usage: $0 migratint [username] [dashboards...]\n"</span><span class="p">;</span>
<span class="nv">has</span> <span class="nv">description</span> <span class="o">=></span> <span class="s">"kibana-int index migration for auth users\n"</span><span class="p">;</span>
<span class="nv">has</span> <span class="nv">ua</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="nn">Mojo::</span><span class="nv">UserAgent</span><span class="o">-></span><span class="k">new</span> <span class="p">};</span>
<span class="k">sub </span><span class="nf">run</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$self</span><span class="p">,</span> <span class="nv">$user</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$config</span> <span class="o">=</span> <span class="nv">$self</span><span class="o">-></span><span class="nv">app</span><span class="o">-></span><span class="nv">config</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$ua</span> <span class="o">=</span> <span class="nv">$self</span><span class="o">-></span><span class="nv">ua</span><span class="p">;</span>
<span class="o">...</span>
<span class="p">}</span>
<span class="mi">1</span><span class="p">;</span>
</code></pre>
</div>
<p>大致就是这样:</p>
<p>继承 <strong>Mojolicious::Command</strong> 类。这样就会有 usage 和 description 两个属性和 run 方法。</p>
<ul>
<li>usage 属性用来在你执行 <code class="highlighter-rouge">script/myapp help mycommand</code> 的时候输出信息;</li>
<li>description 属性用来在你执行 <code class="highlighter-rouge">script/myapp help</code> 罗列全部可用子命令的时候描述该命令的作用;</li>
<li>run 方法是命令的入口函数。命令行后面的参数都会传递给 run 方法。如果你的子命令需要复杂处理,这里依然可以用 <a href="https://metacpan.org/pod/Getopt::Long#Parsing-options-from-an-arbitrary-array">GetOpt::Long</a> 模块中的 <code class="highlighter-rouge">GetOptionsFromArray</code> 函数处理。</li>
</ul>
在 logstash 里使用其他 RubyGems 模块
2014-09-24T00:00:00+08:00
logstash
java
ruby
http://chenlinux.com/2014/09/24/howto-use-custom-rubygem-in-logstash
<p>在开发和使用一些 logstash 自定义插件的时候,几乎不可避免会导入其他 RubyGems 模块 —— 因为都用不上模块的小型处理,直接写在 <em>filters/ruby</em> 插件配置里就够了 —— 这时候,运行 logstash 命令可能会发现一个问题:这个 gem 模块一直是 “no found” 状态。</p>
<p>这其实是因为我们一般是通过 java 命令来运行的 logstash,这时候它回去寻找的 Gem 路径跟我们预计中的是不一致的。</p>
<p>要查看 logstash 运行时实际的 Gem 查找路径,首先要通过 <code class="highlighter-rouge">ps aux</code> 命令确定 ruby 的实际运行方式:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ ps uax|grep logstash
raochenlin 27268 38.0 4.3 3268156 181344 s003 S+ 7:10PM 0:22.36 /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/bin/java -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /Downloads/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar -I/Users/raochenlin/Downloads/logstash-1.4.2/lib /Users/raochenlin/Downloads/logstash-1.4.2/lib/logstash/runner.rb agent -f test.conf
</code></pre>
</div>
<p>看,实际的运行方式应该是:<code class="highlighter-rouge">java -jar logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar -Ilogstash-1.4.2/lib logstash-1.4.2/lib/logstash/runner.rb</code> 这样。</p>
<p>那么我们查看 gem 路径的命令也就知道怎么写了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>java -jar logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar `which gem` env
</code></pre>
</div>
<p>你会看到这样的输出:</p>
<blockquote>
<p>RubyGems Environment:<br />
- RUBYGEMS VERSION: 2.1.9<br />
- RUBY VERSION: 1.9.3 (2014-02-24 patchlevel 392) [java]<br />
- INSTALLATION DIRECTORY: file:/Downloads/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/gems/shared<br />
- RUBY EXECUTABLE: java -jar /Downloads/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar<br />
- EXECUTABLE DIRECTORY: file:/Downloads/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/bin<br />
- SPEC CACHE DIRECTORY: /.gem/specs<br />
- RUBYGEMS PLATFORMS:<br />
- ruby<br />
- universal-java-1.7<br />
- GEM PATHS:<br />
- file:/Downloads/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/gems/shared<br />
- /.gem/jruby/1.9<br />
- GEM CONFIGURATION:<br />
- :update_sources => true<br />
- :verbose => true<br />
- :backtrace => false<br />
- :bulk_threshold => 1000<br />
- “install” => “–no-rdoc –no-ri –env-shebang”<br />
- “update” => “–no-rdoc –no-ri –env-shebang”<br />
- :sources => [“http://ruby.taobao.org/”]<br />
- REMOTE SOURCES:<br />
- http://ruby.taobao.org/<br />
- SHELL PATH:<br />
- /usr/bin<br />
- /bin<br />
- /usr/sbin<br />
- /sbin<br />
- /usr/local/bin</p>
</blockquote>
<p>看到其中的 GEM PATHS 部分,是一个以 <strong>file:</strong> 开头的路径!也就是说,要求所有的 gem 包都打包在这个 jruby-complete-1.7.11.jar 里面才认。</p>
<p>所以我们需要把额外的 gem 包,也加入这个 jar 里:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>jar uf jruby-completa-1.7.11.jar META-INF/jruby.home/lib/ruby/1.9/CUSTOM_RUBY_GEM_LIB
</code></pre>
</div>
<p><em>注:加入 jar 是用的相对路径,所以前面这串目录要提前创建然后复制文件进去。</em></p>
<p>当然,其实还有另一个办法。</p>
<p>让我们返回去再看一次 logstash 的进程,在 jar 后面,还有一个 <code class="highlighter-rouge">-I</code> 参数!所以,其实我们还可以把文件安装在 <code class="highlighter-rouge">logstash-1.4.2/lib</code> 目录下去。</p>
<p>最后,你可能会问:那 <code class="highlighter-rouge">--pluginpath</code> 参数指定的位置可不可以呢?</p>
<p>答案是:也可以。</p>
<p>这个参数指定的位置在 <em>logstash-1.4.2/lib/logstash/agent.rb</em> 中,被加入了 <code class="highlighter-rouge">$LOAD_PATH</code> 中:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">def</span> <span class="nf">configure_plugin_path</span><span class="p">(</span><span class="n">paths</span><span class="p">)</span>
<span class="n">paths</span><span class="p">.</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">path</span><span class="o">|</span>
<span class="k">if</span> <span class="o">!</span><span class="no">Dir</span><span class="p">.</span><span class="nf">exists?</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="nb">warn</span><span class="p">(</span><span class="no">I18n</span><span class="p">.</span><span class="nf">t</span><span class="p">(</span><span class="s2">"logstash.agent.configuration.plugin_path_missing"</span><span class="p">,</span>
<span class="ss">:path</span> <span class="o">=></span> <span class="n">path</span><span class="p">))</span>
<span class="k">end</span>
<span class="n">plugin_glob</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="s2">"logstash"</span><span class="p">,</span> <span class="s2">"{inputs,codecs,filters,outputs}"</span><span class="p">,</span> <span class="s2">"*.rb"</span><span class="p">)</span>
<span class="k">if</span> <span class="no">Dir</span><span class="p">.</span><span class="nf">glob</span><span class="p">(</span><span class="n">plugin_glob</span><span class="p">).</span><span class="nf">empty?</span>
<span class="vi">@logger</span><span class="p">.</span><span class="nf">warn</span><span class="p">(</span><span class="no">I18n</span><span class="p">.</span><span class="nf">t</span><span class="p">(</span><span class="s2">"logstash.agent.configuration.no_plugins_found"</span><span class="p">,</span>
<span class="ss">:path</span> <span class="o">=></span> <span class="n">path</span><span class="p">,</span> <span class="ss">:plugin_glob</span> <span class="o">=></span> <span class="n">plugin_glob</span><span class="p">))</span>
<span class="k">end</span>
<span class="vi">@logger</span><span class="p">.</span><span class="nf">debug</span><span class="p">(</span><span class="s2">"Adding plugin path"</span><span class="p">,</span> <span class="ss">:path</span> <span class="o">=></span> <span class="n">path</span><span class="p">)</span>
<span class="vg">$LOAD_PATH</span><span class="p">.</span><span class="nf">unshift</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p><code class="highlighter-rouge">$LOAD_PATH</code> 是 Ruby 的一个特殊变量,类似于 Perl 的 <code class="highlighter-rouge">@INC</code> 或者 Java 的 <code class="highlighter-rouge">class_path</code> 。在这个数组里的路径下的文件,都可以被 require 导入。</p>
<p>可以运行如下命令查看:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ java -jar logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar -e 'p $LOAD_PATH'
["file:/Users/raochenlin/Downloads/logstash-1.4.2/vendor/jar/rar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/1.9/site_ruby", "file:/Users/raochenlin/Downloads/logstash-1.4.2/vendor/jar/rar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared", "file:/Users/raochenlin/Downloads/logstash-1.4.2/vendor/jar/rar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/1.9"]
</code></pre>
</div>
<p>这三种方式,你喜欢哪种呢?</p>
Kibana 认证鉴权方案
2014-09-23T00:00:00+08:00
logstash
perl
kibana
elasticsearch
mojolicious
http://chenlinux.com/2014/09/23/kibana-auth
<p>Kibana 作为一个纯 JS 项目,一直都没有提供完整的权限控制方面的功能。只是附带了一个 <code class="highlighter-rouge">nginx.conf</code> 做基本的 Basic Auth。社区另外有在 nodejs 上实现的方案,则使用了 CAS 方式做认证。</p>
<p>不过我对这两种方案都不太满意。</p>
<ol>
<li>认证方式太单一,适应性不强;</li>
<li>权限隔离不明确,只是通过修改 <code class="highlighter-rouge">kibana-int</code> 成 <code class="highlighter-rouge">kiban-int-user</code> 来区分不同用户的 dashboard,并不能限制用户对 ES 索引的访问。</li>
</ol>
<p>加上 nodejs 我也不熟,最终在多番考虑后,决定抽一个晚上自己写一版。</p>
<p>最终代码见 <a href="https://github.com/chenryn/kibana">https://github.com/chenryn/kibana</a>。</p>
<h2 id="section">原理和实现</h2>
<ol>
<li>
<p>全站代理和虚拟响应</p>
<p>这里不单通过 config.js 限定了 kibana 默认连接的 Elasticsearch 服务器地址和端口,还拦截伪造了 <code class="highlighter-rouge">/_nodes</code> 请求的 JSON 响应体。伪造的响应中也只包含自己这个带认证的 web 服务器地址和端口。</p>
<p><em>这么做是因为我的 kibana 版本使用的 elasticjs 库比官方新增了 sniff 功能,默认会自动轮训所有 nodes 发送请求。</em></p>
</li>
<li>
<p>新增 <code class="highlighter-rouge">kibana-auth</code> 鉴权索引</p>
<p>在通常的 <code class="highlighter-rouge">kibana-int-user</code> 区分 dashboard 基础上,我新增加 <code class="highlighter-rouge">kibana-auth</code> 索引,专门记录每个用户可以访问的 ES 集群地址和索引前缀。请求会固定代理到指定的 ES 集群上,并且确认是被允许访问的索引。</p>
<p>这样,多个用户通过一个 kibana auth 服务器网址,可以访问多个不同的 ES 集群后端。而同一个 ES 集群后端的索引,也不用担心被其他人访问到。</p>
</li>
<li>
<p><a href="https://metacpan.org/pod/Authen::Simple">Authen::Simple</a> 认证框架</p>
<p>这是 Perl 一个认证框架,支持十多种不同的认证方式。项目里默认采用最简单的 htpasswd 文件记录方式,实际我线上是使用了 LDAP 方式,都没问题。</p>
</li>
</ol>
<h2 id="section-1">部署</h2>
<p>方案采用了 Mojolicious 框架开发,代码少不说,最关键的是 Mojolicious 无额外的 CPAN 模块依赖,这对于不了解 Perl 但是又有 Kibana 权限控制需求的人来说,大大减少了部署方面的麻烦。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>curl -Lk http://cpanmin.us -o /usr/local/bin/cpanm
chmod +x /usr/local/bin/cpanm
cpanm Mojolicious Authen::Simple::Passwd
</code></pre>
</div>
<p>三行命令,就可以完成整个项目的安装需求了。然后运行目录下的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>hypnotoad script/kbnauth
</code></pre>
</div>
<p>就可以通过 80 端口访问这个带有权限控制的 kibana 了。</p>
<p><strong>2015 年 1 月 6 日更新:</strong></p>
<p>目前已经提供了 bundle 方式。有编译环境的可以直接用</p>
<div class="highlighter-rouge"><pre class="highlight"><code>./vendor/bin/carton install --cached
./vendor/bin/carton <span class="nb">exec local</span>/bin/hypnotoad script/kbnauth
</code></pre>
</div>
<h2 id="section-2">权限赋值</h2>
<p>因为 <code class="highlighter-rouge">kibana-auth</code> 结构很简单,kibana 一般又都是内部使用,所以暂时还没做权限控制的管理页面。直接通过命令行方式即可赋权:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>curl -XPOST http://127.0.0.1:9200/kibana-auth/indices/sri -d <span class="s1">'{
"prefix":["logstash-sri","logstash-ops"],
"server":"192.168.0.2:9200"
}'</span>
</code></pre>
</div>
<p>这样,sri 用户,就只能访问 192.168.0.2 集群上的 logstash-sri 或 logstash-ops 开头的日期型索引(即后面可以-YYYY, -YYYY.MM, -YYYY.MM.dd 三种格式)了。</p>
<h2 id="section-3">下一步</h2>
<p>考虑到新方案下各用户都有自己的 <code class="highlighter-rouge">kibana-int-user</code> 索引,已经用着官方 kibana 的用户大批量的 dashboard 有迁移成本,找个时间可能做一个迁移脚本辅助这个事情。</p>
<p>开发完成后,得到了 <a href="http://weibo.com/u/1808998161">@高伟</a> 童鞋的主动尝试和各种 bug 反馈支持,在此表示感谢~也希望我这个方案能帮到更多 kibana 用户。</p>
<p><strong>注:我的 kibana 仓库除了新增的这个 kbnauth 代理认证鉴权功能外,本身在 kibana 分析统计功能上也有一些改进,这方面已经得到多个小伙伴的试用和好评,自认在官方 Kibana v4 版本出来之前,应该会是最好用的版本。欢迎大家下载使用!</strong></p>
<p>新增功能包括:</p>
<ol>
<li>仿 stats 的百分比统计面板(利用 PercentileAggr 接口)</li>
<li>仿 terms 的区间比面板(利用 RangeFacets 接口)</li>
<li>给 bettermap 增强的高德地图支持(利用 leaflet provider 扩展)</li>
<li>给 map 增强的中国地图支持(利用 jvectormap 文件)</li>
<li>给 map 增强的 <code class="highlighter-rouge">term_stats</code> 数据显示(利用 TermStatsFacets 接口)</li>
<li>给 query 增强的请求生成器(利用 getMapping/getFieldMapping 接口和 jQuery.multiSelect 扩展)</li>
<li>仿 terms 的 statisticstrend 面板(利用 TermStatsFacets 接口)</li>
<li>仿 histogram 增强的 multifieldhistogram 面板(可以给不同query定制不同的panel setting,比如设置某个抽样数据 * 1000 倍和另一个全量数据做对比)</li>
<li>仿 histogram 的 valuehistogram 面板(去除了 histogram 面板的 X 轴时间类型数据限制,可以用于做数据概率分布分析)</li>
<li>给 histogram 增强的 threshold 变色功能(利用了 <code class="highlighter-rouge">jquery.flot.threshold</code> 扩展)</li>
<li>单个面板自己的刷新按钮(避免调试的时候全页面刷新的麻烦)</li>
<li>重写 histogram 并增强了 uniq 去重统计模式(利用 CardinalityAggr 接口)</li>
<li>给 terms 增强的自定义脚本化字段聚合功能(利用 scriptField 方法)</li>
<li>给 filterSrv 增强的自定义脚本化过滤器功能,配合上条的点击生成(利用 scriptFilter 接口)</li>
<li>给 table 增强的导出 CSV 功能(利用 filesaver.js)</li>
</ol>
<p>效果截图同样在 <a href="https://github.com/chenryn/kibana/blob/master/README.md">README</a> 里贴出。欢迎试用和反馈!</p>
用 Spark 处理数据导入 Elasticsearch
2014-09-04T00:00:00+08:00
logstash
elasticsearch
spark
scala
http://chenlinux.com/2014/09/04/spark-to-elasticsearch
<p>Logstash 说了这么多。其实运用 Kibana 和 Elasticsearch 不一定需要 logstash,其他各种工具导入的数据都可以。今天就演示一个特别的~用 Spark 来处理导入数据。</p>
<p>首先分别下载 spark 和 elasticsearch-hadoop 的软件包。注意 elasticsearch-hadoop 从最新的 2.1 版开始才带有 spark 支持,所以要下新版:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2-bin-cdh4.tgz
wget http://download.elasticsearch.org/hadoop/elasticsearch-hadoop-2.1.0.Beta1.zip
</code></pre>
</div>
<p>分别解压开后,运行 spark 交互命令行 <code class="highlighter-rouge">ADD_JARS=../elasticsearch-hadoop-2.1.0.Beta1/dist/elasticsearch-spark_2.10-2.1.0.Beta1.jar ./bin/spark-shell</code> 就可以逐行输入 scala 语句测试了。</p>
<p><strong>注意 elasticsearch 不支持 1.6 版本的 java,所以在 MacBook 上还设置了一下 <code class="highlighter-rouge">JAVA_HOME="/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home"</code> 启用自己从 Oracle 下载安装的 1.7 版本的 Java。</strong></p>
<h1 id="section">基础示例</h1>
<p>首先来个最简单的测试,可以展示写入 ES 的用法:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">org.apache.spark.SparkConf</span>
<span class="kn">import</span> <span class="nn">org.elasticsearch.spark._</span>
<span class="c1">// 更多 ES 设置,见<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/2.1.Beta/configuration.html></span>
<span class="n">val</span> <span class="n">conf</span> <span class="o">=</span> <span class="k">new</span> <span class="n">SparkConf</span><span class="o">()</span>
<span class="n">conf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">"es.index.auto.create"</span><span class="o">,</span> <span class="s">"true"</span><span class="o">)</span>
<span class="n">conf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">"es.nodes"</span><span class="o">,</span> <span class="s">"127.0.0.1"</span><span class="o">)</span>
<span class="c1">// 在spark-shell下默认已建立</span>
<span class="c1">// import org.apache.spark.SparkContext </span>
<span class="c1">// import org.apache.spark.SparkContext._</span>
<span class="c1">// val sc = new SparkContext(conf)</span>
<span class="n">val</span> <span class="n">numbers</span> <span class="o">=</span> <span class="n">Map</span><span class="o">(</span><span class="s">"one"</span> <span class="o">-></span> <span class="mi">1</span><span class="o">,</span> <span class="s">"two"</span> <span class="o">-></span> <span class="mi">2</span><span class="o">,</span> <span class="s">"three"</span> <span class="o">-></span> <span class="mi">3</span><span class="o">)</span>
<span class="n">val</span> <span class="n">airports</span> <span class="o">=</span> <span class="n">Map</span><span class="o">(</span><span class="s">"OTP"</span> <span class="o">-></span> <span class="s">"Otopeni"</span><span class="o">,</span> <span class="s">"SFO"</span> <span class="o">-></span> <span class="s">"San Fran"</span><span class="o">)</span>
<span class="n">sc</span><span class="o">.</span><span class="na">makeRDD</span><span class="o">(</span><span class="n">Seq</span><span class="o">(</span><span class="n">numbers</span><span class="o">,</span> <span class="n">airports</span><span class="o">)).</span><span class="na">saveToEs</span><span class="o">(</span><span class="s">"spark/docs"</span><span class="o">)</span>
</code></pre>
</div>
<p>这就 OK 了。尝试访问一下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ curl '127.0.0.1:9200/spark/docs/_search?q=*'
</code></pre>
</div>
<p>返回结果如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="nt">"took"</span><span class="p">:</span><span class="mi">66</span><span class="p">,</span><span class="nt">"timed_out"</span><span class="p">:</span><span class="kc">false</span><span class="p">,</span><span class="nt">"_shards"</span><span class="p">:{</span><span class="nt">"total"</span><span class="p">:</span><span class="mi">5</span><span class="p">,</span><span class="nt">"successful"</span><span class="p">:</span><span class="mi">5</span><span class="p">,</span><span class="nt">"failed"</span><span class="p">:</span><span class="mi">0</span><span class="p">},</span><span class="nt">"hits"</span><span class="p">:{</span><span class="nt">"total"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"max_score"</span><span class="p">:</span><span class="mf">1.0</span><span class="p">,</span><span class="nt">"hits"</span><span class="p">:[{</span><span class="nt">"_index"</span><span class="p">:</span><span class="s2">"spark"</span><span class="p">,</span><span class="nt">"_type"</span><span class="p">:</span><span class="s2">"docs"</span><span class="p">,</span><span class="nt">"_id"</span><span class="p">:</span><span class="s2">"BwNJi8l2TmSRTp42GhDmww"</span><span class="p">,</span><span class="nt">"_score"</span><span class="p">:</span><span class="mf">1.0</span><span class="p">,</span><span class="w"> </span><span class="nt">"_source"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nt">"one"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"two"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="nt">"three"</span><span class="p">:</span><span class="mi">3</span><span class="p">}},{</span><span class="nt">"_index"</span><span class="p">:</span><span class="s2">"spark"</span><span class="p">,</span><span class="nt">"_type"</span><span class="p">:</span><span class="s2">"docs"</span><span class="p">,</span><span class="nt">"_id"</span><span class="p">:</span><span class="s2">"7f7ar-9kSb6WEiLS8ROUCg"</span><span class="p">,</span><span class="nt">"_score"</span><span class="p">:</span><span class="mf">1.0</span><span class="p">,</span><span class="w"> </span><span class="nt">"_source"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nt">"OTP"</span><span class="p">:</span><span class="s2">"Otopeni"</span><span class="p">,</span><span class="nt">"SFO"</span><span class="p">:</span><span class="s2">"San Fran"</span><span class="p">}}]}}</span><span class="w">
</span></code></pre>
</div>
<h1 id="section-1">文件处理</h1>
<p>下一步,我们看如何读取文件和截取字段。scala 也提供了正则和捕获的方法:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">var</span> <span class="n">text</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="na">textFile</span><span class="o">(</span><span class="s">"/var/log/system.log"</span><span class="o">)</span>
<span class="n">var</span> <span class="n">Pattern</span> <span class="o">=</span> <span class="s">"""(\w{3}\s+\d{1,2} \d{2}:\d{2}:\d{2}) (\S+) (\S+)\[(\d+)\]: (.+)"""</span><span class="o">.</span><span class="na">r</span>
<span class="n">var</span> <span class="n">entries</span> <span class="o">=</span> <span class="n">text</span><span class="o">.</span><span class="na">map</span> <span class="o">{</span>
<span class="k">case</span> <span class="n">Pattern</span><span class="o">(</span><span class="n">timestamp</span><span class="o">,</span> <span class="n">host</span><span class="o">,</span> <span class="n">program</span><span class="o">,</span> <span class="n">pid</span><span class="o">,</span> <span class="n">message</span><span class="o">)</span> <span class="o">=></span> <span class="n">Map</span><span class="o">(</span><span class="s">"timestamp"</span> <span class="o">-></span> <span class="n">timestamp</span><span class="o">,</span> <span class="s">"host"</span> <span class="o">-></span> <span class="n">host</span><span class="o">,</span> <span class="s">"program"</span> <span class="o">-></span> <span class="n">program</span><span class="o">,</span> <span class="s">"pid"</span> <span class="o">-></span> <span class="n">pid</span><span class="o">,</span> <span class="s">"message"</span> <span class="o">-></span> <span class="n">message</span><span class="o">)</span>
<span class="k">case</span> <span class="o">(</span><span class="n">line</span><span class="o">)</span> <span class="o">=></span> <span class="n">Map</span><span class="o">(</span><span class="s">"message"</span> <span class="o">-></span> <span class="n">line</span><span class="o">)</span>
<span class="o">}</span>
<span class="n">entries</span><span class="o">.</span><span class="na">saveToEs</span><span class="o">(</span><span class="s">"spark/docs"</span><span class="o">)</span>
</code></pre>
</div>
<p>这里示例写了两个 <code class="highlighter-rouge">case</code> ,因为 Mac 上的 “system.log” 不知道用的什么 syslog 协议,有些在 <code class="highlighter-rouge">[pid]</code> 后面居然还有一个 <code class="highlighter-rouge">(***)</code> 才是 <code class="highlighter-rouge">:</code>。正好就可以用这个来示例如果匹配失败的情况如何处理。不加这个默认 <code class="highlighter-rouge">case</code> 的话,匹配失败的就直接报错不会存进 <code class="highlighter-rouge">entries</code> 对象了。</p>
<p><strong>注意:<code class="highlighter-rouge">.textFile</code> 不是 scala 标准的读取文件函数,而是 sparkContext 对象的方法,返回的是 RDD 对象(包括后面的 <code class="highlighter-rouge">.map</code> 返回的也是新的 RDD 对象)。所以后面就不用再 <code class="highlighter-rouge">.makeRDD</code> 了。</strong></p>
<h1 id="section-2">网络数据</h1>
<p>Spark 还有 Spark streaming 子项目,用于从其他网络协议读取数据,比如 flume,kafka,zeromq 等。官网上有一个配合 <code class="highlighter-rouge">nc -l</code> 命令的示例程序。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">org.apache.spark.streaming._</span>
<span class="n">val</span> <span class="n">ssc</span> <span class="o">=</span> <span class="k">new</span> <span class="n">StreamingContext</span><span class="o">(</span><span class="n">sc</span><span class="o">,</span> <span class="n">Seconds</span><span class="o">(</span><span class="mi">1</span><span class="o">))</span>
<span class="n">val</span> <span class="n">lines</span> <span class="o">=</span> <span class="n">ssc</span><span class="o">.</span><span class="na">socketTextStream</span><span class="o">(</span><span class="s">"localhost"</span><span class="o">,</span> <span class="mi">9999</span><span class="o">)</span>
<span class="o">...</span>
<span class="n">ssc</span><span class="o">.</span><span class="na">start</span><span class="o">()</span>
<span class="n">ssc</span><span class="o">.</span><span class="na">awaitTermination</span><span class="o">()</span>
</code></pre>
</div>
<p>有时间我会继续尝试 Spark 其他功能。</p>
山寨一个 Splunk 的 source 上下文查看功能
2014-08-29T00:00:00+08:00
logstash
elasticsearch
splunk
http://chenlinux.com/2014/08/29/implement-source-context-function-for-elk
<p>跟很多朋友在聊 elk stack 的时候,都会不知不觉的开始跟 Splunk 做对比。最常见的两个抱怨就是:Splunk 的搜索构建语法 比 Kibana 方便,以及 Splunk 搜索出来的消息可以通过点击 <code class="highlighter-rouge">Source</code> 按钮查看其原始日志中的前后几条日志。</p>
<p><img src="/images/uploads/splunk-source-context.jpg" alt="splunk source context" /></p>
<p>平心而论,这个上下文查找的功能确实在排错过程中非常有用。但是在 elk 里却不那么容易实现,原因是:</p>
<p><strong>elasticsearch 是一个分布式项目,其索引的 <code class="highlighter-rouge">_id</code> 默认使用的是 UUID 方式生成的随机字符串,你没法根据 UUID 来判断数据的先后。</strong></p>
<p><code class="highlighter-rouge">LogStash::Outputs::Elasticsearch</code> 提供了让你指定 <code class="highlighter-rouge">_id</code> 内容的选项,但是在集群环境下,你很难自己搞定一个全局自增 ID。</p>
<p><em>相反,虽然我不知道 splunk 的数据存储的内部实现,但是就他昂贵的报价来说,基本只见过单机案例。就单机而言,自增 id 太轻松了</em></p>
<p>所以,从原理上来说,就很难实现一个通用的 elk 版上下文查看功能。</p>
<p>不过我们缩小一下使用场景,却未必不能自己山寨一个对自己可用的办法来。</p>
<p>假设我们一个最常见的场景,就是从各 web 服务器上收集不同日志到中心。那么这时候,通过 <code class="highlighter-rouge">%{host}</code> 和 <code class="highlighter-rouge">%{path}</code> 的 “AND” 过滤,我们就可以把范围缩小到一个单一的文件内容里。所以,我们只需要能够搞定这个文件的自增 id 就够了!</p>
<h2 id="logstashconf-">logstash.conf 示例</h2>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">input</span> <span class="p">{</span>
<span class="n">file</span> <span class="p">{</span>
<span class="n">path</span> <span class="o">=></span> <span class="p">[</span><span class="s2">"/var/log/*.log"</span><span class="p">]</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">filter</span> <span class="p">{</span>
<span class="n">ruby</span> <span class="p">{</span>
<span class="n">init</span> <span class="o">=></span> <span class="s1">'@incr={}'</span>
<span class="n">code</span> <span class="o">=></span> <span class="s2">"key = event['host']+event['path']
if @incr.has_key?(key)
@incr[key] += 1
else
@incr[key] = 1
end
event['lineno'] = @incr[key]"</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">output</span> <span class="p">{</span>
<span class="n">elasticsearch</span> <span class="p">{</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<h2 id="curl-">上下文查询 curl 示例</h2>
<p>使用上面的配置运行起来 logstash 之后,假设我们现在搜到一条 syslog 日志,其 <code class="highlighter-rouge">lineno</code> 是 20,那么查看它的前后 5 条记录的 curl 命令就是:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="err">curl</span><span class="w"> </span><span class="err">-XPOST</span><span class="w"> </span><span class="err">'http://localhost:</span><span class="mi">9200</span><span class="err">/logstash</span><span class="mf">-2014.08</span><span class="err">.</span><span class="mi">29</span><span class="err">/_search?pretty=</span><span class="mi">1</span><span class="err">'</span><span class="w"> </span><span class="err">-d</span><span class="w"> </span><span class="err">'</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nt">"query"</span><span class="p">:{</span><span class="w">
</span><span class="nt">"range"</span><span class="p">:{</span><span class="w">
</span><span class="nt">"lineno"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"gt"</span><span class="p">:</span><span class="mi">15</span><span class="p">,</span><span class="w">
</span><span class="nt">"lte"</span><span class="p">:</span><span class="mi">25</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"filter"</span><span class="p">:{</span><span class="w">
</span><span class="nt">"term"</span><span class="p">:{</span><span class="w">
</span><span class="nt">"host.raw"</span><span class="p">:</span><span class="s2">"raochenlindeMacBook-Air.local"</span><span class="p">,</span><span class="w">
</span><span class="nt">"path.raw"</span><span class="p">:</span><span class="s2">"/var/log/system.log"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"sort"</span><span class="p">:[{</span><span class="nt">"lineno"</span><span class="p">:</span><span class="s2">"asc"</span><span class="p">}],</span><span class="w">
</span><span class="nt">"fields"</span><span class="p">:[</span><span class="s2">"message"</span><span class="p">],</span><span class="w">
</span><span class="nt">"size"</span><span class="p">:</span><span class="mi">10</span><span class="w">
</span><span class="p">}</span><span class="err">'</span><span class="w">
</span></code></pre>
</div>
<p>得到的结果是:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nt">"took"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w">
</span><span class="nt">"timed_out"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nt">"_shards"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="p">,</span><span class="w">
</span><span class="nt">"successful"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="p">,</span><span class="w">
</span><span class="nt">"failed"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"total"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">10</span><span class="p">,</span><span class="w">
</span><span class="nt">"max_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nt">"hits"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2014.08.29"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"ILkv4oZOQRGXkH5nxjPT6Q"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nt">"fields"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"Aug 29 23:34:44 raochenlindeMacBook-Air.local stunnel[304]: LOG5[4391727104]: Service [sproxy] accepted connection from 127.0.0.1:52673"</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"sort"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mi">16</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2014.08.29"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"frRzVZUDQr-dkRog9LEypQ"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nt">"fields"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"Aug 29 23:34:44 raochenlindeMacBook-Air.local stunnel[304]: LOG5[4391727104]: s_connect: connected 50.116.12.155:65080"</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"sort"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mi">17</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2014.08.29"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"fQ50VrbuSfy6AmhNOaHpFg"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nt">"fields"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"Aug 29 23:34:44 raochenlindeMacBook-Air.local stunnel[304]: LOG5[4391727104]: Service [sproxy] connected remote server from 192.168.0.102:52674"</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"sort"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mi">18</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2014.08.29"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Bpza8x6gSQi3OFRfAz3vPA"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nt">"fields"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"Aug 29 23:35:23 raochenlindeMacBook-Air.local stunnel[304]: LOG5[4391882752]: Service [sproxy] accepted connection from 127.0.0.1:52710"</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"sort"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mi">19</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2014.08.29"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"I7SQ4o-aSr--em1WXO0y0A"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nt">"fields"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"Aug 29 23:35:24 raochenlindeMacBook-Air.local stunnel[304]: LOG5[4391882752]: s_connect: connected 50.116.12.155:65080"</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"sort"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mi">20</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2014.08.29"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"POLq7XA_QVe6E5f9cP9V-w"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nt">"fields"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"Aug 29 23:35:24 raochenlindeMacBook-Air.local stunnel[304]: LOG5[4391882752]: Service [sproxy] connected remote server from 192.168.0.102:52711"</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"sort"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2014.08.29"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"sXCLVr7URu-2uKhcOP3wjA"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nt">"fields"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"Aug 29 23:35:35 raochenlindeMacBook-Air.local stunnel[304]: LOG5[4391882752]: Connection closed: 0 byte(s) sent to SSL, 0 byte(s) sent to socket"</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"sort"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mi">22</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2014.08.29"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"3wxxElNuS7OgyvjSm8CQfg"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nt">"fields"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"Aug 29 23:36:25 raochenlindeMacBook-Air.local stunnel[304]: LOG5[4391571456]: Connection closed: 2825 byte(s) sent to SSL, 2407 byte(s) sent to socket"</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"sort"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mi">23</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2014.08.29"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"xdsiB1cmRpagWiMxtAjMzQ"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nt">"fields"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"Aug 29 23:36:52 raochenlindeMacBook-Air.local stunnel[304]: LOG5[4391493632]: Connection closed: 1109 byte(s) sent to SSL, 583 byte(s) sent to socket"</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"sort"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mi">24</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash-2014.08.29"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logs"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"mLScPMbwTzSPMz9WqOPXlw"</span><span class="p">,</span><span class="w">
</span><span class="nt">"_score"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nt">"fields"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"message"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"Aug 29 23:36:52 raochenlindeMacBook-Air.local stunnel[304]: LOG5[4391571456]: Service [sproxy] accepted connection from 127.0.0.1:52719"</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"sort"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mi">25</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
</div>
<p>没错,这就是我们想要的结果了!</p>
<p><strong>注释</strong></p>
<p>这里两个要点:</p>
<ul>
<li>自增 id 为啥不用行号,因为 <code class="highlighter-rouge">LogStash::Inputs::File</code> 实现是通过 <code class="highlighter-rouge">File.seek</code> 和 <code class="highlighter-rouge">File.sysread(16394)</code> 完成的,这种时候 <code class="highlighter-rouge">File.lineno</code> 永远都是 0。获取真的行号很困难。</li>
<li>自增 id 为什么不指定成 <code class="highlighter-rouge">_id</code> 而是另外存字段,因为 <code class="highlighter-rouge">_id</code> 是特殊字段,要求在一个 <code class="highlighter-rouge">_index/_type</code> 里是唯一的。我们对 logstash 的使用一般情况下都是多个 host 内容存在同一个 <code class="highlighter-rouge">_index/_type</code> 下,会发生重复的(重复写入 <code class="highlighter-rouge">_id</code> 相同的数据等同于 <code class="highlighter-rouge">update</code> 操作)。</li>
</ul>
<h2 id="section">延伸</h2>
<p>数据如何通过 kibana 展示,则是另外一个层面的内容。有时间可能我会也做一下。</p>
<p>非 input/file 方式的其他场景,只要你能通过 event 中其他字段确定出来源唯一,都可以采用这个方式做。</p>
用 ES 的 RangeFacets 接口实现一个查看区间占比的 Kibana 面板
2014-08-18T00:00:00+08:00
logstash
kibana
elasticsearch
javascript
http://chenlinux.com/2014/08/18/intro-range-facet-and-implement-panel-for-it
<p>公司用 kibana 的同事提出一个需求,希望查看响应时间在不同区间内占比的饼图。第一想法是用 1.3.0 新加的 percentile rank aggregation 接口。不过仔细想想,其实并不合适 —— 这个接口目的是计算固定的 <code class="highlighter-rouge">[0 TO $value]</code> 的比例。不同的区间反而还得自己做减法来计算。稍微查了一下,更适合的做法是专门的 range aggregation。考虑到 kibana 内大多数还是用 facet 接口,这里也沿用:<a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-facets-range-facet.html">http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-facets-range-facet.html</a>。</p>
<p>range facet 本身的使用非常简单,就像官网示例那样,直接 curl 命令就可以完成调试:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>curl -XPOST http://localhost:9200/logstash-2014.08.18/_search?pretty=1 -d '{
"query" : {
"match_all" : {}
},
"facets" : {
"range1" : {
"range" : {
"field" : "resp_ms",
"ranges" : [
{ "to" : 100 },
{ "from" : 101, "to" : 500 },
{ "from" : 500 }
]
}
}
}
}'
</code></pre>
</div>
<p>不过在 kibana 里,我们就不要再自己拼 JSON 发请求了 —— 虽然之前我实现 percentile panel 的时候就是这么做的 —— 前两天合并了 github 上一个 commit 后,现在可以用高版本的 elastic.js 了,所以我也把原来用原生 <code class="highlighter-rouge">$http.post</code> 方法写的 percentile panel 用 elastic.js 对象重写了。</p>
<p>elastic.js 关于 range facet 的文档见:<a href="http://docs.fullscale.co/elasticjs/ejs.RangeFacet.html">http://docs.fullscale.co/elasticjs/ejs.RangeFacet.html</a></p>
<p>因为 range facet 本身比较简单,所以 RangeFacet 对象支持的方法也比较少。一个 <code class="highlighter-rouge">addRange</code> 方法添加 ranges 数组,一个 <code class="highlighter-rouge">field</code> 方法添加 field 名称,就没了。</p>
<p>所以这个新 panel 的实现,更复杂的地方在如何让 range 范围值支持自定义填写。这一部分借鉴了同样是前两天合并的 github 上另一个第三方面板 multifieldhistogram 的写法。</p>
<p>另一个需要注意的地方是饼图出来以后,单击饼图区域,自动生成的 <code class="highlighter-rouge">filterSrv</code> 内容。一般的面板这里都是 <code class="highlighter-rouge">terms</code> 类型的 <code class="highlighter-rouge">filterSrv</code>,传递的是面板的 label 值。而我们这里 label 值显然不是 ES 有效的 terms 语法,还好 <code class="highlighter-rouge">filterSrv</code> 有 <code class="highlighter-rouge">range</code> 类型(histogram 面板的 <code class="highlighter-rouge">time</code> 类型的 <code class="highlighter-rouge">filterSrv</code> 是在 daterange 基础上实现的),所以稍微修改就可以了。</p>
<p>最终效果如下:</p>
<p><img src="https://github.com/chenryn/kibana/raw/master/src/img/chenryn_img/range-panel.jpg" alt="" /></p>
<p>面板的属性界面如下:</p>
<p><img src="https://github.com/chenryn/kibana/raw/master/src/img/chenryn_img/range-setting.jpg" alt="" /></p>
<p>代码已经上传到我个人 fork 的 kibana 项目里:<a href="https://github.com/chenryn/kibana.git">https://github.com/chenryn/kibana.git</a></p>
<p><em>我这个 kibana 里已经综合了 8 个第三方面板或重要修改。在官方年底推出 4.0 版本之间,自觉还是值得推荐给大家用的。具体修改说明和效果图见 README。</em></p>
Kibana 动态仪表板的使用
2014-07-28T00:00:00+08:00
logstash
kibana
http://chenlinux.com/2014/07/28/dynamic-dashboard-for-kibana
<p>半年前,Kibana3.4 版刚出来的时候,曾经在官方博客上描述了一个新功能,当时我的翻译见:<a href="/2014/01/15/kibana3-milestone4-20131105/">【翻译】Kibana3 里程碑 4</a>。</p>
<p>今天我实际使用了一下这个新功能,感觉还是蛮有用的,单独拿出来记录一下用法和一些没在之前文章里提到的细节。</p>
<h2 id="section">使用方法</h2>
<p>使用方法其实在官方描述里已经比较清楚了。就是在原本的 <code class="highlighter-rouge">http://127.0.0.1:9292/#/dashboard/file/logstash.json</code> 地址后面,再加上请求参数 <code class="highlighter-rouge">?query=***</code> 即可。</p>
<h2 id="section-1">注意事项</h2>
<p>看起来好像太过简单,不过用起来其实还是有点注意事项的:</p>
<ul>
<li>Kibana 目前不支持对保存在 Elasticsearch 中的 dashboard 做这个事情。</li>
</ul>
<p>所以一定得保存成 <code class="highlighter-rouge">yourname.json</code> 文件放入 <code class="highlighter-rouge">app/dashboards/</code> 目录里才行。</p>
<ul>
<li>静态的 JSON 文件其实是利用模板技术。</li>
</ul>
<p>所以直接导出得到的 JSON 文件还不能直接起作用。需要稍微做一点修改。</p>
<p>你可以打开默认可用的 <code class="highlighter-rouge">logstash.json</code> 文件,看看有什么奇特的地方,没错,就是下面这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>"query": "{<span>{</span>ARGS.query || '*'}}"
</code></pre>
</div>
<p>而你自己保存下来的 JSON,这里都会是具体的数据。所以,要让自己的 JSON 布局也支持动态仪表板的话,按照这个写法也都加上 <code class="highlighter-rouge">ARGS.query</code> 就好了!</p>
<p>从 logstash.json 里还可以看到,除了 <code class="highlighter-rouge">?query=</code> 以外,其实还支持 <code class="highlighter-rouge">from=</code> 参数,默认是 <code class="highlighter-rouge">24h</code>。</p>
<ul>
<li>query 参数的特殊字符问题。</li>
</ul>
<p>比如我之前在搜索框里输入的 querystring 是这样的:<code class="highlighter-rouge">type:mweibo_action AND urlpath:"/2/statuses/friends_timeline.json"</code> 。</p>
<p>那么实际用的时候,如果写成这样一个 url:<code class="highlighter-rouge">http://127.0.0.1:9292/#/dashboard/file/logstash.json?query=type:mweibo_action AND urlpath:"/2/statuses/friends_timeline.json"</code>,实际是不对的。我一度怀疑是不是 urlpath 里的 <code class="highlighter-rouge">/</code> 导致的问题,后来发现,其实是 <code class="highlighter-rouge">"</code> 在进 JSON 文件模板变量替换的时候给当做只是字符串赋值引号的作用,就不再作为字符串本身传递给 Elasticsearch 作为请求内容本身了。<strong>所以需要用 <code class="highlighter-rouge">\</code> 给 <code class="highlighter-rouge">"</code> 做转义。</strong></p>
<p>(这里一定要有 <code class="highlighter-rouge">"</code> 的原因是,ES 的 querystring 里,<code class="highlighter-rouge">field:/regex/</code> 是正则匹配搜索的语法,刚好 url 也是以 <code class="highlighter-rouge">/</code> 开头的)</p>
<p>所以可用的 url 应该是:<code class="highlighter-rouge">http://127.0.0.1:9292/#/dashboard/file/logstash.json?query=type:mweibo_action AND urlpath:\"/2/statuses/friends_timeline.json\"</code>!</p>
<p>经过 url_encode 之后就变成了:<code class="highlighter-rouge">http://127.0.0.1:9292/#/dashboard/file/logstash.json?query=type:mweibo_action%20AND%20urlpath:%5C%22%2F2%2Fstatuses%2Ffriends_timeline.json%5C%22</code></p>
<p>这样就可以了!</p>
<ul>
<li>用 JSON 的局限。</li>
</ul>
<p>动态仪表板其实有两种用法,这里只用到了 <code class="highlighter-rouge">file/logstash.json</code> 静态文件方式,这种方式只支持一个 query 条件,也没有太多的附加参数支持。而 <code class="highlighter-rouge">script/logstash.js</code> 方式,支持多个 query 条件,以及 index、pattern、interval、timefield 等更多的参数选项。</p>
<p>当然,研究一下 angularjs 的用法,给 JSON 文件里也加上 <code class="highlighter-rouge">ARGS.query</code> 的 <code class="highlighter-rouge">split</code> 方法,也不算太难。</p>
在 MacBook 上使用 PDL 绘图
2014-07-27T00:00:00+08:00
perl
zabbix
pdl
python
numpy
macbook
http://chenlinux.com/2014/07/27/using-pdl-on-macbook
<p>之前在 Linux 服务器上使用 PDL,主要是一些矩阵函数,这次准备在个人电脑上使用 PDL,尤其是本身的绘图功能,其一目的就是导出 zabbix 中存储的监控数据,通过 PDL 绘图观察其季节性分布情况。</p>
<p>不过在使用的时候,发现在 MacBook 上跑 PDL 还是有点上手难度的。和 pylab 不同,PDL 是使用了 X11 的,而 MacBook 最新的版本里,X11 已经不再是自带的了。所以需要单独去下载 <a href="https://www.macupdate.com/app/mac/26593/xquartz">XQuartz</a> 安装包来提供 X11 支持。</p>
<p>安装好了 XQuartz 以后,再安装 PDL::Graphics:: 名字空间下的几个模块就好办了。</p>
<ul>
<li>PDL::Graphics::Simple</li>
<li>PDL::Graphics::Gnuplot</li>
<li>PDL::Graphics::PGPLOT</li>
<li>PDL::Graphics::Prima</li>
</ul>
<p>另外还有 PDL::Graphics::PLplot 等,不过通过 <code class="highlighter-rouge">port install plplot</code> 安装的 plplot 没有 header 文件,所以 PDL::Graphics::PLplot 是安装不上的,既然前面已经有了不少,这里也就不再追求自己下载 plplot 源代码来安装了。</p>
<p>PDL::Graphics::Simple 是 《PDL Book》开篇第一个示例就使用的模块,其实际就是按顺序尝试加载 <code class="highlighter-rouge">::Gnuplot</code>、<code class="highlighter-rouge">::PGPLOT</code>、<code class="highlighter-rouge">::PLplot</code> 和 <code class="highlighter-rouge">::Prima</code>。所以,保证有一个可用就好了。</p>
<p>不过在我的 air 上实际的效果来看,perldl 命令在使用 子进程跟 gnuplot 交互的时候<strong>非常非常非常的慢!</strong></p>
<p>好了,现在就可以运行程序了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/env perl</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">utf8</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">feature</span> <span class="s">':5.16'</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Path::</span><span class="nv">Tiny</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">YAML</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">PDL</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">PDL::Graphics::</span><span class="nv">PGPLOT</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Zabbix2::</span><span class="nv">API</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$config</span> <span class="o">=</span> <span class="nv">Load</span><span class="p">(</span> <span class="nv">path</span><span class="p">(</span><span class="s">'config.yml'</span><span class="p">)</span><span class="o">-></span><span class="nv">slurp</span> <span class="p">);</span>
<span class="k">my</span> <span class="nv">$zbconf</span> <span class="o">=</span> <span class="nv">$config</span><span class="o">-></span><span class="p">{</span><span class="s">'zabbix'</span><span class="p">};</span>
<span class="k">my</span> <span class="nv">$zabbix</span> <span class="o">=</span>
<span class="nn">Zabbix2::</span><span class="nv">API</span><span class="o">-></span><span class="k">new</span><span class="p">(</span>
<span class="nv">server</span> <span class="o">=></span> <span class="s">"http://$zbconf->{'addr'}/zabbix/api_jsonrpc.php"</span> <span class="p">);</span>
<span class="nb">eval</span> <span class="p">{</span>
<span class="nv">$zabbix</span><span class="o">-></span><span class="nv">login</span><span class="p">(</span>
<span class="nv">user</span> <span class="o">=></span> <span class="nv">$zbconf</span><span class="o">-></span><span class="p">{</span><span class="s">'user'</span><span class="p">},</span>
<span class="nv">password</span> <span class="o">=></span> <span class="nv">$zbconf</span><span class="o">-></span><span class="p">{</span><span class="s">'pass'</span><span class="p">}</span>
<span class="p">);</span>
<span class="p">};</span>
<span class="k">if</span> <span class="p">(</span><span class="vg">$@</span><span class="p">)</span> <span class="p">{</span> <span class="nb">die</span> <span class="s">'could not authenticate'</span> <span class="p">}</span>
<span class="k">my</span> <span class="nv">$items</span> <span class="o">=</span> <span class="nv">$zabbix</span><span class="o">-></span><span class="nv">fetch</span><span class="p">(</span>
<span class="s">'Item'</span><span class="p">,</span>
<span class="nv">params</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">groupids</span> <span class="o">=></span> <span class="mi">21</span><span class="p">,</span>
<span class="nv">hostids</span> <span class="o">=></span> <span class="mi">11036</span><span class="p">,</span>
<span class="nv">graphids</span> <span class="o">=></span> <span class="mi">1824829</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">);</span>
<span class="k">for</span> <span class="k">my</span> <span class="nv">$item</span> <span class="p">(</span><span class="nv">@$items</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">say</span> <span class="nv">$item</span><span class="o">-></span><span class="nv">data</span><span class="o">-></span><span class="p">{</span><span class="s">'name'</span><span class="p">};</span>
<span class="k">my</span> <span class="nv">$itemid</span> <span class="o">=</span> <span class="nv">$item</span><span class="o">-></span><span class="nv">data</span><span class="o">-></span><span class="p">{</span><span class="s">'itemid'</span><span class="p">};</span>
<span class="nv">say</span> <span class="nv">$itemid</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$sitems</span> <span class="o">=</span> <span class="nv">$zabbix</span><span class="o">-></span><span class="nv">fetch_single</span><span class="p">(</span>
<span class="s">'Item'</span><span class="p">,</span>
<span class="nv">params</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">itemids</span> <span class="o">=></span> <span class="nv">$itemid</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">);</span>
<span class="k">my</span> <span class="nv">$pdl</span> <span class="o">=</span> <span class="nv">pdl</span><span class="p">(</span><span class="nb">map</span> <span class="p">{</span><span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="nv">value</span><span class="p">}}</span> <span class="nv">@</span><span class="p">{</span> <span class="nv">$sitems</span><span class="o">-></span><span class="nv">history</span><span class="p">(</span> <span class="nv">time_from</span> <span class="o">=></span> <span class="nb">time</span><span class="p">()</span> <span class="o">-</span> <span class="mi">24</span> <span class="o">*</span> <span class="mi">3600</span> <span class="p">)</span> <span class="p">});</span>
<span class="nv">bin</span><span class="p">(</span><span class="nv">hist</span><span class="p">(</span><span class="nv">$pdl</span><span class="p">));</span>
<span class="k">my</span> <span class="nv">$lline</span> <span class="o">=</span> <span class="nv">pct</span><span class="p">(</span><span class="nv">$pdl</span><span class="p">,</span> <span class="mf">0.25</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$uline</span> <span class="o">=</span> <span class="nv">pct</span><span class="p">(</span><span class="nv">$pdl</span><span class="p">,</span> <span class="mf">0.75</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$low</span> <span class="o">=</span> <span class="mi">2</span> <span class="o">*</span> <span class="nv">$lline</span> <span class="o">-</span> <span class="nv">$uline</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$up</span> <span class="o">=</span> <span class="mi">2</span> <span class="o">*</span> <span class="nv">$uline</span> <span class="o">-</span> <span class="nv">$lline</span><span class="p">;</span>
<span class="nv">say</span> <span class="nv">$pdl</span><span class="o">-></span><span class="nv">where</span><span class="p">(</span><span class="nv">$pdl</span><span class="o">></span><span class="nv">$up</span> <span class="o">|</span> <span class="nv">$pdl</span><span class="o"><</span><span class="nv">$low</span><span class="p">);</span>
<span class="p">}</span>
</code></pre>
</div>
<p>这里使用了 <a href="https://metacpan.org/pod/Zabbix2::API">Zabbix2::API</a> 模块,相对比 <a href="http://blog.zabbix.com/getting-started-with-zabbix-api/1381/">zabbix 官方博客示例</a>直接使用 <a href="https://metacpan.org/pod/JSON::RPC">JSON::RPC</a> 模块,以及 python 的 pyzabbix 模块来说,Zabbix2::API 模块封装的非常好,history 是作为 item 对象的属性出现,而不是单独再请求一次 <code class="highlighter-rouge">history.get</code>;item 的 name 等属性也非常友好和有用。</p>
<p>另外,不知道为什么,使用 pyzabbix 模块就一直无法正常使用,而自己写 requests 和 json 却没问题。上面的 perl 脚本用 python 改写就是下面这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#!/usr/bin/env python</span>
<span class="s">"""
Read item history from zabbix, and plot as histogram
"""</span>
<span class="kn">import</span> <span class="nn">matplotlib</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">matplotlib.mlab</span> <span class="kn">as</span> <span class="nn">mlab</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="kn">as</span> <span class="nn">plt</span>
<span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="n">ZABBIX_URI</span> <span class="o">=</span> <span class="s">'http://test.zabbix.com/zabbix/api_jsonrpc.php'</span>
<span class="n">ZABBIX_USR</span> <span class="o">=</span> <span class="s">'user'</span>
<span class="n">ZABBIX_PWD</span> <span class="o">=</span> <span class="s">'pass'</span>
<span class="n">HOURS</span> <span class="o">=</span> <span class="mi">24</span> <span class="o">*</span> <span class="mi">1</span>
<span class="k">def</span> <span class="nf">zabbixLogin</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="n">passwd</span><span class="p">):</span>
<span class="n">params</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'user'</span><span class="p">:</span><span class="n">user</span><span class="p">,</span>
<span class="s">'password'</span><span class="p">:</span><span class="n">passwd</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">zabbixCall</span><span class="p">(</span><span class="s">'user.login'</span><span class="p">,</span> <span class="n">params</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">zabbixCall</span><span class="p">(</span><span class="n">method</span><span class="o">=</span><span class="s">''</span><span class="p">,</span> <span class="n">params</span><span class="o">=</span><span class="p">{},</span> <span class="n">auth</span><span class="o">=</span><span class="s">''</span><span class="p">):</span>
<span class="n">data</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'jsonrpc'</span><span class="p">:</span><span class="s">'2.0'</span><span class="p">,</span>
<span class="s">'method'</span><span class="p">:</span><span class="n">method</span><span class="p">,</span>
<span class="s">'params'</span><span class="p">:</span><span class="n">params</span><span class="p">,</span>
<span class="s">'id'</span><span class="p">:</span><span class="mi">1</span>
<span class="p">}</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">auth</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">data</span><span class="p">[</span><span class="s">'auth'</span><span class="p">]</span> <span class="o">=</span> <span class="n">auth</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="n">ZABBIX_URI</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">data</span><span class="p">),</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="s">'content-type'</span><span class="p">:</span><span class="s">'application/json-rpc'</span><span class="p">})</span>
<span class="k">return</span> <span class="n">r</span><span class="o">.</span><span class="n">json</span><span class="p">()[</span><span class="s">'result'</span><span class="p">]</span>
<span class="n">authId</span> <span class="o">=</span> <span class="n">zabbixLogin</span><span class="p">(</span><span class="n">ZABBIX_USR</span><span class="p">,</span> <span class="n">ZABBIX_PWD</span><span class="p">)</span>
<span class="n">params</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'groupids'</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span>
<span class="s">'hostids'</span><span class="p">:</span><span class="mi">11036</span><span class="p">,</span>
<span class="s">'graphids'</span><span class="p">:</span><span class="mi">1824829</span>
<span class="p">}</span>
<span class="n">items</span> <span class="o">=</span> <span class="n">zabbixCall</span><span class="p">(</span><span class="s">'item.get'</span><span class="p">,</span> <span class="n">params</span><span class="p">,</span> <span class="n">authId</span><span class="p">)</span>
<span class="n">begin</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">mktime</span><span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span><span class="o">.</span><span class="n">timetuple</span><span class="p">())</span> <span class="o">-</span> <span class="mi">3600</span> <span class="o">*</span> <span class="n">HOURS</span>
<span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">items</span><span class="p">:</span>
<span class="n">params</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'output'</span><span class="p">:</span><span class="s">'extend'</span><span class="p">,</span>
<span class="s">'history'</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span>
<span class="s">'itemids'</span><span class="p">:</span><span class="n">item</span><span class="p">[</span><span class="s">'itemid'</span><span class="p">],</span>
<span class="s">'time_from'</span><span class="p">:</span><span class="n">begin</span>
<span class="p">}</span>
<span class="n">ret</span> <span class="o">=</span> <span class="n">zabbixCall</span><span class="p">(</span><span class="s">'history.get'</span><span class="p">,</span> <span class="n">params</span><span class="p">,</span> <span class="n">authId</span><span class="p">)</span>
<span class="n">history</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="nb">float</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="s">'value'</span><span class="p">]),</span> <span class="n">ret</span><span class="p">)</span>
<span class="n">v</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">history</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">hist</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">bins</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span> <span class="n">normed</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s">'item: '</span> <span class="o">+</span> <span class="n">item</span><span class="p">[</span><span class="s">'itemid'</span><span class="p">])</span>
<span class="c"># lline = numpy.percentile(v, 25)</span>
<span class="c"># uline = numpy.percentile(v, 75)</span>
<span class="c"># low = 2 * lline - uline</span>
<span class="c"># up = 2 * uline - lline</span>
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">boxplot</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">sym</span><span class="o">=</span><span class="s">'+'</span><span class="p">,</span> <span class="n">notch</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s">'item: '</span> <span class="o">+</span> <span class="n">item</span><span class="p">[</span><span class="s">'itemid'</span><span class="p">])</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</code></pre>
</div>
Rex::Test::Spec 模块
2014-07-08T00:00:00+08:00
perl
devops
serverspec
rex
testing
http://chenlinux.com/2014/07/08/rex-test-spec
<p>上篇说了 serverspec 工具,我一直对 Rspec 的语法蛮有好感的,于是昨晚花了点时间模仿这个给 Rex 写了个类似的工具,叫 Rex::Test::Spec,源代码地址见:<a href="https://github.com/chenryn/Rex--Test--Spec">https://github.com/chenryn/Rex--Test--Spec</a>。</p>
<p>语法大概是这样的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nn">Rex::Test::</span><span class="nv">Spec</span><span class="p">;</span>
<span class="nv">describe</span> <span class="s">"Nginx Test"</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">context</span> <span class="nv">run</span><span class="p">(</span><span class="s">"nginx -t"</span><span class="p">),</span> <span class="s">"nginx.conf testing"</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">like</span> <span class="nv">its</span><span class="p">(</span><span class="s">'stdout'</span><span class="p">),</span> <span class="sx">qr/ok/</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">context</span> <span class="nv">file</span><span class="p">(</span><span class="s">"~/.ssh/id_rsa"</span><span class="p">),</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">is</span> <span class="nv">its</span><span class="p">(</span><span class="s">'ensure'</span><span class="p">),</span> <span class="s">'file'</span><span class="p">;</span>
<span class="nv">is</span> <span class="nv">its</span><span class="p">(</span><span class="s">'mode'</span><span class="p">),</span> <span class="s">'0600'</span><span class="p">;</span>
<span class="nv">like</span> <span class="nv">its</span><span class="p">(</span><span class="s">'content'</span><span class="p">),</span> <span class="sx">qr/name\@email\.com/</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">context</span> <span class="nv">file</span><span class="p">(</span><span class="s">"/data"</span><span class="p">),</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">is</span> <span class="nv">its</span><span class="p">(</span><span class="s">'ensure'</span><span class="p">),</span> <span class="s">'directory'</span><span class="p">;</span>
<span class="nv">is</span> <span class="nv">its</span><span class="p">(</span><span class="s">'owner'</span><span class="p">),</span> <span class="s">'www'</span><span class="p">;</span>
<span class="nv">is</span> <span class="nv">its</span><span class="p">(</span><span class="s">'mounted_on'</span><span class="p">),</span> <span class="s">'/dev/sdb1'</span><span class="p">;</span>
<span class="nv">isnt</span> <span class="nv">its</span><span class="p">(</span><span class="s">'writable'</span><span class="p">);</span>
<span class="p">};</span>
<span class="nv">context</span> <span class="nv">service</span><span class="p">(</span><span class="s">"nginx"</span><span class="p">),</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">is</span> <span class="nv">its</span><span class="p">(</span><span class="s">'ensure'</span><span class="p">),</span> <span class="s">'running'</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">context</span> <span class="nv">pkg</span><span class="p">(</span><span class="s">"nginx"</span><span class="p">),</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">is</span> <span class="nv">its</span><span class="p">(</span><span class="s">'ensure'</span><span class="p">),</span> <span class="s">'present'</span><span class="p">;</span>
<span class="nv">is</span> <span class="nv">its</span><span class="p">(</span><span class="s">'version'</span><span class="p">),</span> <span class="s">'1.5.8'</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">context</span> <span class="nv">cron</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">like</span> <span class="nv">its</span><span class="p">(</span><span class="s">'www'</span><span class="p">),</span> <span class="s">'logrotate'</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">context</span> <span class="nv">gateway</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">is</span> <span class="nv">it</span><span class="p">,</span> <span class="s">'192.168.0.1'</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">context</span> <span class="nv">group</span><span class="p">(</span><span class="s">'www'</span><span class="p">),</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">ok</span> <span class="nv">its</span><span class="p">(</span><span class="s">'ensure'</span><span class="p">);</span>
<span class="p">};</span>
<span class="nv">context</span> <span class="nv">port</span><span class="p">(</span><span class="mi">80</span><span class="p">),</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">is</span> <span class="nv">its</span><span class="p">(</span><span class="s">'bind'</span><span class="p">),</span> <span class="s">'0.0.0.0'</span><span class="p">;</span>
<span class="nv">is</span> <span class="nv">its</span><span class="p">(</span><span class="s">'proto'</span><span class="p">),</span> <span class="s">'tcp'</span><span class="p">;</span>
<span class="nv">is</span> <span class="nv">its</span><span class="p">(</span><span class="s">'command'</span><span class="p">),</span> <span class="s">'nginx'</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">context</span> <span class="nv">process</span><span class="p">(</span><span class="s">'nginx'</span><span class="p">),</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">like</span> <span class="nv">its</span><span class="p">(</span><span class="s">'command'</span><span class="p">),</span> <span class="sx">qr(nginx -c /etc/nginx.conf)</span><span class="p">;</span>
<span class="nv">ok</span> <span class="nv">its</span><span class="p">(</span><span class="s">'mem'</span><span class="p">)</span> <span class="o">></span> <span class="mi">1024</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">context</span> <span class="nv">routes</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">is_deeply</span> <span class="nv">its</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> <span class="p">{</span>
<span class="nv">destination</span> <span class="o">=></span> <span class="nv">$dest</span><span class="p">,</span>
<span class="nv">gateway</span> <span class="o">=></span> <span class="nv">$gw</span><span class="p">,</span>
<span class="nv">genmask</span> <span class="o">=></span> <span class="nv">$genmask</span><span class="p">,</span>
<span class="nv">flags</span> <span class="o">=></span> <span class="nv">$flags</span><span class="p">,</span>
<span class="nv">mss</span> <span class="o">=></span> <span class="nv">$mss</span><span class="p">,</span>
<span class="nv">irtt</span> <span class="o">=></span> <span class="nv">$irtt</span><span class="p">,</span>
<span class="nv">iface</span> <span class="o">=></span> <span class="nv">$iface</span><span class="p">,</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="nv">context</span> <span class="nv">sysctl</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">is</span> <span class="nv">its</span><span class="p">(</span><span class="s">'vm.swapiness'</span><span class="p">),</span> <span class="mi">1</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">context</span> <span class="nv">user</span><span class="p">(</span><span class="s">'www'</span><span class="p">),</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">ok</span> <span class="nv">its</span><span class="p">(</span><span class="s">'ensure'</span><span class="p">);</span>
<span class="nv">is</span> <span class="nv">its</span><span class="p">(</span><span class="s">'home'</span><span class="p">),</span> <span class="s">'/var/www/html'</span><span class="p">;</span>
<span class="nv">is</span> <span class="nv">its</span><span class="p">(</span><span class="s">'shell'</span><span class="p">),</span> <span class="s">'/sbin/nologin'</span><span class="p">;</span>
<span class="nv">is_deeply</span> <span class="nv">its</span><span class="p">(</span><span class="s">'belong_to'</span><span class="p">),</span> <span class="p">[</span><span class="s">'www'</span><span class="p">,</span> <span class="s">'nogroup'</span><span class="p">];</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="nv">done_testing</span><span class="p">;</span>
</code></pre>
</div>
<p>从 Rspec 学来的 context/describe/it/its 语法,保留了 Test::More 的 is/like/is_deeply/done_testing 语法。</p>
<p>这里把 Test::More 里导入的指令都重载了,因为把 context 指令后面的资源类型通过 <code class="highlighter-rouge">local $msg</code> 变量传递过来,就可以显示出来每个 <code class="highlighter-rouge">its</code> 测试是什么资源类型的了。因为这个原因,指令导出的时候就没法用 <code class="highlighter-rouge">Exporter</code> 模块,因为 Exporter 里的 import 函数没有 <code class="highlighter-rouge">no strict;no warnings</code>。所以得自己写 import 函数导出。</p>
<p>具体的资源类型,第一次学习了一下 AUTOLOAD 的用法。还是蛮好玩的~</p>
<p>因为我是在 Mac 上写的代码,而 Rex 本身不怎么支持 Darwin 平台,所以源码里就测了一下 run 指令可用。欢迎大家帮忙补齐其他指令的测试用例,以及如何在 Rex 的 task 里通过 SSH 方式远程做这些测试(公司平台也没法让我做这个 SSH 测试)。</p>
Serverspec 工具介绍
2014-06-13T00:00:00+08:00
devops
rspec
ruby
puppet
http://chenlinux.com/2014/06/13/serverspec-intro
<p>去年曾经写过一篇<a href="http://chenlinux.com/2013/01/10/rspec-puppet-intro">文章</a>里提到做 puppet 的测试,用的是 <a href="http://rspec-puppet.com">rspec-puppet</a> 工具。不过这个工具的作用只是能确保在 Puppet Master 上你撰写的 .pp 文件可以按照你的预期正常编译完毕,并不代表真实的节点就是按照这个状态维护的。所以今天介绍另一个工具,Serverspec,它拥有和 rspec-puppet 类似的语法(都是 Rspec 衍生品),同时又是真的 SSH 到远程主机上去做测试!官网见:<a href="http://serverspec.org">http://serverspec.org</a>。</p>
<p>安装直接通过 <code class="highlighter-rouge">gem install serverspec</code> 方式即可完成。然后通过 <code class="highlighter-rouge">serverspec-init</code> 命令可以创建处理来一个测试模板:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>.
├── Rakefile
└── spec
├── 10.4.1.21
│ └── puppet_spec.rb
├── spec_helper.rb
</code></pre>
</div>
<p>文件其实非常简单,所以之后就可以不用命令,自己创建目录和测试文件好了。目录以远端主机 IP 命名,测试文件叫 <code class="highlighter-rouge">foobar_spec.rb</code> 也没关系,反正在 Rakefile 里是通过 <code class="highlighter-rouge">spec/*/*_spec.rb</code> 载入的。</p>
<p>下面是我写的这个 <code class="highlighter-rouge">puppet_spec.rb</code> 实例:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nb">require</span> <span class="s1">'spec_helper'</span>
<span class="n">describe</span> <span class="s2">"system"</span> <span class="k">do</span>
<span class="c1"># TODO: bonding</span>
<span class="n">context</span> <span class="n">interface</span><span class="p">(</span><span class="s1">'eth2'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">have_ipv4_address</span><span class="p">(</span><span class="s2">"192.168.0.200"</span><span class="p">)</span> <span class="p">}</span>
<span class="n">its</span><span class="p">(</span><span class="ss">:speed</span><span class="p">)</span> <span class="p">{</span> <span class="n">should</span> <span class="n">eq</span> <span class="mi">1000</span> <span class="p">}</span>
<span class="k">end</span>
<span class="n">context</span> <span class="n">file</span><span class="p">(</span><span class="s1">'/data'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">be_mounted</span><span class="p">.</span><span class="nf">with</span><span class="p">(</span> <span class="ss">:type</span> <span class="o">=></span> <span class="s1">'ext4'</span> <span class="p">)</span> <span class="p">}</span>
<span class="k">end</span>
<span class="n">context</span> <span class="n">linux_kernel_parameter</span><span class="p">(</span><span class="s1">'vm.swappiness'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">its</span><span class="p">(</span><span class="ss">:value</span><span class="p">)</span> <span class="p">{</span> <span class="n">should</span> <span class="n">eq</span> <span class="mi">0</span> <span class="p">}</span>
<span class="k">end</span>
<span class="n">context</span> <span class="n">yumrepo</span><span class="p">(</span><span class="s1">'epel'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">exist</span> <span class="p">}</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should_not</span> <span class="n">be_enabled</span> <span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">describe</span> <span class="s2">"puppetmaster"</span> <span class="k">do</span>
<span class="n">context</span> <span class="n">group</span><span class="p">(</span><span class="s1">'puppet'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">exist</span> <span class="p">}</span>
<span class="k">end</span>
<span class="n">context</span> <span class="n">user</span><span class="p">(</span><span class="s1">'puppet'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">exist</span> <span class="p">}</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">belong_to_group</span> <span class="s1">'puppet'</span> <span class="p">}</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should_not</span> <span class="n">have_login_shell</span> <span class="s1">'/bin/sh'</span> <span class="p">}</span>
<span class="k">end</span>
<span class="n">context</span> <span class="n">package</span><span class="p">(</span><span class="s1">'puppet'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">be_installed</span><span class="p">.</span><span class="nf">by</span><span class="p">(</span><span class="s1">'gem'</span><span class="p">).</span><span class="nf">with_version</span><span class="p">(</span><span class="s1">'3.6.1'</span><span class="p">)</span> <span class="p">}</span>
<span class="k">end</span>
<span class="n">context</span> <span class="n">package</span><span class="p">(</span><span class="s1">'nginx'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">be_installed</span> <span class="p">}</span>
<span class="k">end</span>
<span class="n">context</span> <span class="n">service</span><span class="p">(</span><span class="s1">'nginx'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">be_enabled</span> <span class="p">}</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">be_running</span> <span class="p">}</span>
<span class="k">end</span>
<span class="sx">%w[8140 18140]</span><span class="p">.</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">port</span><span class="o">|</span>
<span class="n">context</span> <span class="n">port</span><span class="p">(</span><span class="n">port</span><span class="p">)</span> <span class="k">do</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">be_listening</span> <span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">context</span> <span class="n">file</span><span class="p">(</span><span class="s1">'/etc/nginx/sites-enabled/puppet'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">be_linked_to</span> <span class="s1">'/etc/puppet/webui/ngx_puppetmaster.conf'</span> <span class="p">}</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">be_readable</span><span class="p">.</span><span class="nf">by_user</span><span class="p">(</span><span class="s1">'nobody'</span><span class="p">)</span> <span class="p">}</span>
<span class="n">its</span><span class="p">(</span><span class="ss">:content</span><span class="p">)</span> <span class="p">{</span> <span class="n">should</span> <span class="n">match</span> <span class="sr">/\n\s*server 127.0.0.1:18140;/</span> <span class="p">}</span>
<span class="k">end</span>
<span class="n">context</span> <span class="n">command</span><span class="p">(</span><span class="s2">"nginx -t"</span><span class="p">)</span> <span class="k">do</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">return_stderr</span> <span class="sr">/ok/</span> <span class="p">}</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">return_exit_status</span> <span class="mi">0</span> <span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">describe</span> <span class="n">process</span><span class="p">(</span><span class="s1">'rrdcached'</span><span class="p">)</span> <span class="k">do</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">be_running</span> <span class="p">}</span>
<span class="n">its</span><span class="p">(</span><span class="ss">:args</span><span class="p">)</span> <span class="p">{</span> <span class="n">should</span> <span class="n">match</span> <span class="sr">/-j \/omd\/sites\/cdn\/var\/rrdcached/</span> <span class="p">}</span>
<span class="k">end</span>
</code></pre>
</div>
<p>基本上可以说跟 puppet 最常用的几个类型对应的测试就都在上面展示了。此外,Serverspec 与时俱进,还提供了 <code class="highlighter-rouge">cgroup</code> 和 <code class="highlighter-rouge">lxc</code> 的测试器。这里就没写了。</p>
<p>这里有个注意到的问题就是网卡速度那里,是不支持测试 bonding 网卡的。它 ssh 上去后其实就是执行 ethtool 命令,ethtool 命令获取不到,自然也就没法测试,肯定会报测试失败。</p>
<p>另一个问题就是文件内容匹配那块,虽然文档示例里用了 <code class="highlighter-rouge">/^begin/</code> 但是实测这个会把整个文本读成一个大字符串来匹配,所以单行的开头不能用 <code class="highlighter-rouge">^</code> 而是用 <code class="highlighter-rouge">\n</code> 来做。</p>
<p>正常情况下,写完测试用例,就可以运行 <code class="highlighter-rouge">rake spec</code> 命令跑测试了。不过熟悉我的朋友都知道人人网这边服务器都是统一通过 Kerberos 认证来管理权限的,而 各种语言的 SSH 模块默认都不太支持 krb5。所以我这还需要先解决 Serverspec 的 krb5 支持问题。</p>
<p>感谢 <a href="http://weibo.com/u/1653644220">@懒桃儿吃桃儿</a> 童鞋贡献的<a href="https://github.com/Lax/net-ssh-kerberos">模块</a>,部署过程如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ git clone https://github.com/Lax/net-ssh-kerberos.git
$ pushd net-ssh-kerberos
$ gem build net-ssh-kerberos.gemspec
$ gem install net-ssh-krb-0.3.0.gem
$ popd
$ diff spec/spec_helper.rb spec/spec_helper.rb.orig
4,5d3
< require 'rubygems'
< require 'net/ssh/kerberos'
29d26
< options[:auth_methods] = ["gssapi-with-mic"]
</code></pre>
</div>
<p>模块文档上说可以通过 Gemfile 配合 <code class="highlighter-rouge">Bundler.require</code> 指令直接运行,我测试自己写脚本的话确实没有问题,但是融合到 <code class="highlighter-rouge">spec_helper.rb</code> 里就不行,所以只能自行编译安装,然后通过 rubygems 模块来加载了。</p>
<p>最后,就可以看到下面这样的输出了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ rake spec
/usr/bin/ruby -S rspec spec/10.4.1.21/nginx_spec.rb
.......................
Finished in 9.99 seconds
23 examples, 0 failures
</code></pre>
</div>
用 Redis 做分布式 DNS/HTTP 检测汇总系统
2014-06-13T00:00:00+08:00
perl
monitor
redis
anyevent
http://chenlinux.com/2014/06/13/anyevent-redis-for-dns-check
<p>一年前搞的一套小脚本,今天翻博客发现没发过,现在发上来好了。主要背景是这样:考虑到有 DNS 和 HTTP 劫持需要监控,但是很多 DNS 服务器对非本区域本运营商的来源请求是拒绝做出响应的,所以得把监控点分散到各地去。其实做这个事情用 nagios 的分布式就足够了,不过如果想做即时触发的紧急任务,就算在 nagios 页面上点击立刻执行,到返回全部结果也得有一阵子。所以选择了自己写一套分布式的异步系统。</p>
<p>中控端脚本如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/perl</span>
<span class="k">use</span> <span class="nn">Modern::</span><span class="nv">Perl</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">AnyEvent</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">AnyEvent::Redis::</span><span class="nv">RipeRedis</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Storable</span> <span class="sx">qw/freeze thaw/</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">YAML::</span><span class="nv">Syck</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">utf8</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$area</span> <span class="o">=</span> <span class="nv">$ARGV</span><span class="p">[</span><span class="mi">0</span><span class="p">];</span>
<span class="k">my</span> <span class="nv">$domain</span> <span class="o">=</span> <span class="s">'fmn.xnimg.cn'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$master</span> <span class="o">=</span> <span class="s">'10.4.1.21'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$cv</span> <span class="o">=</span> <span class="nv">AnyEvent</span><span class="o">-></span><span class="nv">condvar</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$redis</span> <span class="o">=</span> <span class="nn">AnyEvent::Redis::</span><span class="nv">RipeRedis</span><span class="o">-></span><span class="k">new</span><span class="p">(</span>
<span class="nv">host</span> <span class="o">=></span> <span class="nv">$master</span><span class="p">,</span>
<span class="nv">port</span> <span class="o">=></span> <span class="mi">6379</span><span class="p">,</span>
<span class="nv">encoding</span> <span class="o">=></span> <span class="s">'utf8'</span><span class="p">,</span>
<span class="p">);</span>
<span class="k">my</span> <span class="nv">$dnslist</span> <span class="o">=</span> <span class="nv">LoadFile</span><span class="p">(</span><span class="s">"DNS.yml"</span><span class="p">);</span>
<span class="k">for</span> <span class="k">my</span> <span class="nv">$isp</span> <span class="p">(</span> <span class="nb">sort</span> <span class="nb">keys</span> <span class="nv">%$dnslist</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span> <span class="nb">defined</span> <span class="nv">$area</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">next</span> <span class="k">unless</span> <span class="nb">defined</span> <span class="nv">$dnslist</span><span class="o">-></span><span class="p">{</span><span class="nv">$isp</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">$area</span><span class="p">};</span>
<span class="nv">say</span> <span class="nv">$area</span><span class="p">,</span> <span class="nv">$isp</span><span class="p">,</span> <span class="nb">join</span> <span class="s">", "</span><span class="p">,</span> <span class="nv">@</span><span class="p">{</span> <span class="nv">$dnslist</span><span class="o">-></span><span class="p">{</span><span class="nv">$isp</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">$area</span><span class="p">}</span> <span class="p">};</span>
<span class="k">my</span> <span class="nv">$data</span> <span class="o">=</span> <span class="nv">freeze</span><span class="p">({</span> <span class="nv">domain</span> <span class="o">=></span> <span class="nv">$domain</span><span class="p">,</span> <span class="nv">dnslist</span> <span class="o">=></span> <span class="nv">$dnslist</span><span class="o">-></span><span class="p">{</span><span class="nv">$isp</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">$area</span><span class="p">}</span> <span class="p">});</span>
<span class="nv">$redis</span><span class="o">-></span><span class="nv">publish</span><span class="p">(</span> <span class="s">'task'</span><span class="p">,</span> <span class="nv">$data</span> <span class="p">);</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="k">for</span> <span class="k">my</span> <span class="nv">$list</span> <span class="p">(</span> <span class="nb">sort</span> <span class="nb">keys</span> <span class="nv">%</span><span class="p">{</span> <span class="nv">$dnslist</span><span class="o">-></span><span class="p">{</span><span class="nv">$isp</span><span class="p">}</span> <span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$data</span> <span class="o">=</span> <span class="nv">freeze</span><span class="p">({</span> <span class="nv">domain</span> <span class="o">=></span> <span class="nv">$domain</span><span class="p">,</span> <span class="nv">dnslist</span> <span class="o">=></span> <span class="nv">$dnslist</span><span class="o">-></span><span class="p">{</span><span class="nv">$isp</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">$list</span><span class="p">}</span> <span class="p">});</span>
<span class="nv">$cv</span><span class="o">-></span><span class="nv">begin</span><span class="p">;</span>
<span class="nv">$redis</span><span class="o">-></span><span class="nv">publish</span><span class="p">(</span> <span class="s">'task'</span><span class="p">,</span> <span class="nv">$data</span> <span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nv">$redis</span><span class="o">-></span><span class="nv">subscribe</span><span class="p">(</span>
<span class="sx">qw( report )</span><span class="p">,</span>
<span class="p">{</span>
<span class="nv">on_done</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$ch_name</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$subs_num</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">print</span> <span class="s">"Subscribed: $ch_name. Active: $subs_num\n"</span><span class="p">;</span>
<span class="p">},</span>
<span class="nv">on_message</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$ch_name</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$msg</span> <span class="o">=</span> <span class="nv">thaw</span><span class="p">(</span> <span class="nb">shift</span> <span class="p">);</span>
<span class="nb">printf</span> <span class="s">"%s A %s @%s in %s got %s length %s\n"</span><span class="p">,</span> <span class="nv">$domain</span><span class="p">,</span> <span class="nv">$msg</span><span class="o">-></span><span class="p">{</span><span class="nv">ip</span><span class="p">},</span> <span class="nv">$msg</span><span class="o">-></span><span class="p">{</span><span class="nv">dns</span><span class="p">},</span> <span class="nv">$msg</span><span class="o">-></span><span class="p">{</span><span class="nb">local</span><span class="p">},</span> <span class="nv">$msg</span><span class="o">-></span><span class="p">{</span><span class="nv">status</span><span class="p">},</span> <span class="nv">$msg</span><span class="o">-></span><span class="p">{</span><span class="nv">len</span><span class="p">};</span>
<span class="nv">$cv</span><span class="o">-></span><span class="nv">end</span><span class="p">;</span>
<span class="p">},</span>
<span class="nv">on_error</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">print</span> <span class="nv">@_</span><span class="p">;</span>
<span class="p">},</span>
<span class="p">}</span>
<span class="p">);</span>
<span class="nv">$cv</span><span class="o">-></span><span class="nb">recv</span><span class="p">;</span>
</code></pre>
</div>
<p>分布在各地的客户端脚本如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/perl</span>
<span class="k">use</span> <span class="nn">Modern::</span><span class="nv">Perl</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">AnyEvent</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">AnyEvent::</span><span class="nv">Socket</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">AnyEvent::</span><span class="nv">DNS</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">AnyEvent::Redis::</span><span class="nv">RipeRedis</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">AnyEvent::</span><span class="nv">HTTP</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Storable</span> <span class="sx">qw/freeze thaw/</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Digest::</span><span class="nv">MD5</span> <span class="sx">qw/md5_hex/</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">utf8</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$master</span> <span class="o">=</span> <span class="s">'10.4.1.21'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$local</span> <span class="o">=</span> <span class="s">'192.168.0.2'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$cv</span> <span class="o">=</span> <span class="nv">AnyEvent</span><span class="o">-></span><span class="nv">condvar</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$redisr</span> <span class="o">=</span> <span class="nn">AnyEvent::Redis::</span><span class="nv">RipeRedis</span><span class="o">-></span><span class="k">new</span><span class="p">(</span>
<span class="nv">host</span> <span class="o">=></span> <span class="nv">$master</span><span class="p">,</span>
<span class="nv">port</span> <span class="o">=></span> <span class="mi">6379</span><span class="p">,</span>
<span class="nv">encoding</span> <span class="o">=></span> <span class="s">'utf8'</span><span class="p">,</span>
<span class="p">);</span>
<span class="k">my</span> <span class="nv">$redisp</span> <span class="o">=</span> <span class="nn">AnyEvent::Redis::</span><span class="nv">RipeRedis</span><span class="o">-></span><span class="k">new</span><span class="p">(</span>
<span class="nv">host</span> <span class="o">=></span> <span class="nv">$master</span><span class="p">,</span>
<span class="nv">port</span> <span class="o">=></span> <span class="mi">6379</span><span class="p">,</span>
<span class="nv">encoding</span> <span class="o">=></span> <span class="s">'utf8'</span><span class="p">,</span>
<span class="p">);</span>
<span class="nv">$redisr</span><span class="o">-></span><span class="nv">subscribe</span><span class="p">(</span>
<span class="s">'task'</span><span class="p">,</span>
<span class="p">{</span>
<span class="nv">on_done</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$ch_name</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$subs_num</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">print</span> <span class="s">"Subscribed: $ch_name. Active: $subs_num\n"</span><span class="p">;</span>
<span class="p">},</span>
<span class="nv">on_message</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$ch_name</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$msg</span> <span class="o">=</span> <span class="nv">thaw</span><span class="p">(</span><span class="nb">shift</span><span class="p">);</span>
<span class="k">for</span> <span class="k">my</span> <span class="nv">$dns</span> <span class="p">(</span> <span class="nv">@</span><span class="p">{</span> <span class="nv">$msg</span><span class="o">-></span><span class="p">{</span><span class="nv">dnslist</span><span class="p">}</span> <span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">resolv</span><span class="p">(</span> <span class="nv">$dns</span><span class="p">,</span> <span class="nv">$msg</span><span class="o">-></span><span class="p">{</span><span class="nv">domain</span><span class="p">}</span> <span class="p">);</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="nv">on_error</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$err_msg</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$err_code</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">print</span> <span class="s">"Error: ($err_code) $err_msg\n"</span><span class="p">;</span>
<span class="p">},</span>
<span class="p">}</span>
<span class="p">);</span>
<span class="nv">$cv</span><span class="o">-></span><span class="nb">recv</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">resolv</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$dns</span><span class="p">,</span> <span class="nv">$domain</span> <span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">return</span> <span class="k">unless</span> <span class="nv">$dns</span> <span class="o">=~</span> <span class="sr">m/^\d+/</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$resolver</span> <span class="o">=</span>
<span class="nn">AnyEvent::</span><span class="nv">DNS</span><span class="o">-></span><span class="k">new</span><span class="p">(</span> <span class="nv">server</span> <span class="o">=></span> <span class="p">[</span> <span class="nn">AnyEvent::Socket::</span><span class="nv">parse_address</span> <span class="nv">$dns</span> <span class="p">],</span> <span class="p">);</span>
<span class="nv">$resolver</span><span class="o">-></span><span class="nv">resolve</span><span class="p">(</span>
<span class="s">"$domain"</span> <span class="o">=></span> <span class="s">'a'</span><span class="p">,</span>
<span class="k">sub </span><span class="p">{</span>
<span class="nv">httptest</span><span class="p">(</span><span class="nv">$dns</span><span class="p">,</span> <span class="nv">$domain</span><span class="p">,</span> <span class="nv">$_</span><span class="o">-></span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span> <span class="k">for</span> <span class="nv">@_</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">httptest</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$dns</span><span class="p">,</span> <span class="nv">$domain</span><span class="p">,</span> <span class="nv">$ip</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$url</span> <span class="o">=</span> <span class="s">"http://$domain/10k.html"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$begin</span> <span class="o">=</span> <span class="nb">time</span><span class="p">;</span>
<span class="nv">http_get</span> <span class="nv">$url</span><span class="p">,</span> <span class="nv">proxy</span> <span class="o">=></span> <span class="p">[</span><span class="nv">$ip</span><span class="p">,</span> <span class="mi">80</span><span class="p">],</span> <span class="nv">want_body_handle</span> <span class="o">=></span> <span class="mi">1</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$hdl</span><span class="p">,</span> <span class="nv">$hdr</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$port</span><span class="p">,</span> <span class="nv">$peer</span><span class="p">)</span> <span class="o">=</span> <span class="nn">AnyEvent::Socket::</span><span class="nv">unpack_sockaddr</span> <span class="nb">getpeername</span> <span class="nv">$hdl</span><span class="o">-></span><span class="p">{</span><span class="s">'fh'</span><span class="p">};</span>
<span class="k">my</span> <span class="nv">$data</span> <span class="o">=</span> <span class="nv">freeze</span><span class="p">(</span> <span class="p">{</span> <span class="nv">dns</span> <span class="o">=></span> <span class="nv">$dns</span><span class="p">,</span> <span class="nv">status</span> <span class="o">=></span> <span class="nv">$hdr</span><span class="o">-></span><span class="p">{</span><span class="nv">Status</span><span class="p">},</span> <span class="nb">local</span> <span class="o">=></span> <span class="nv">$local</span><span class="p">,</span> <span class="nv">ip</span> <span class="o">=></span> <span class="nv">$peer</span><span class="p">,</span> <span class="nv">len</span> <span class="o">=></span> <span class="nv">$hdr</span><span class="o">-></span><span class="p">{</span><span class="s">'content-length'</span><span class="p">}</span> <span class="p">}</span> <span class="p">);</span>
<span class="nv">$redisp</span><span class="o">-></span><span class="nv">publish</span><span class="p">(</span><span class="s">'report'</span><span class="p">,</span> <span class="nv">$data</span><span class="p">);</span>
<span class="p">};</span>
<span class="p">}</span>
</code></pre>
</div>
<p>这里需要单独建立两个 <code class="highlighter-rouge">$redisr</code> 和 <code class="highlighter-rouge">$redisp</code> ,因为前一个已经用来 subscribe 之后就不能同时用于 publish 了,会报错。从理解上这是个很扯淡的事情,不过实际运行结果就是如此。。。</p>
Rex 简明手册
2014-06-12T00:00:00+08:00
devops
perl
puppet
rex
http://chenlinux.com/2014/06/12/rex-usage
<p>Rex 是 Perl 编写的基于 SSH 链接的集群配置管理系统,语法上类似 Puppet DSL。官网中文版见 <a href="http://rex.perl-china.com">http://rex.perl-china.com</a> 。本文仅为本人在部门 Wiki 上编写的简介性文档。</p>
<h2 id="section">常用命令参数</h2>
<p>rex 命令参数很多,不过因为我们的环境是 krb 认证的,所以有些参数只能写在 Rexfile 里。所以一般固定在存放了 Rexfile 的 /etc/puppet/webui 下执行命令,很多配置就自动加载了。那么还需要用到的命令参数基本就只有下面几个:</p>
<p>-Tv: 查看当前 Rexfile 里定义了哪些 Task 任务,以及服务器组。</p>
<p>-H: 指定 Task 将在哪些 Host 上执行。这里比较方便的地方是支持 <code class="highlighter-rouge">10.5.16.[95..110]</code> 这样的写法。</p>
<p>-G: 指定 Task 将在哪些 Group 上执行。Group 的定义方式很多,Rex 默认支持的有直接在 Rexfile 里通过 group 指令指定,通过 ini 配置文件设定等等。目前我是实现了一个 <code class="highlighter-rouge">groups_db</code> 指令,来从我们的 sqlite 里获取。<code class="highlighter-rouge">groups_db('cdnbj::nginx')</code> 就会自动生成一个名叫 ‘cdnbj::nginx’ 的服务器组,包括 cdnbj 里所有部署了 nginx 的服务器。</p>
<p>-e: 指定一个临时任务。通常是’say run “ipconfig”‘这样的简单命令形式。如果需要复杂逻辑,还是在 Rexfile 里书写 Task。</p>
<p>-q:指定运行日志级别,有 -q 和 -qq。</p>
<p>-d:指定运行日志级别,有 -d 和 -dd。</p>
<h2 id="rexfile-">Rexfile 介绍</h2>
<p>参数设置部分:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">set</span> <span class="nv">connection</span> <span class="o">=></span> <span class="s">"OpenSSH"</span><span class="p">;</span>
<span class="nv">user</span> <span class="s">"root"</span><span class="p">;</span>
<span class="nv">krb5_auth</span><span class="p">;</span>
<span class="nv">parallelism</span> <span class="mi">10</span><span class="p">;</span>
</code></pre>
</div>
<p>这四行指定采用 kerberos 认证,并发 10 个进程执行 ssh 命令。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">desc</span> <span class="s">"install puppet agent"</span><span class="p">;</span>
<span class="nv">task</span> <span class="s">"puppet_install"</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="p">}</span>
<span class="nv">before</span> <span class="s">"puppet_install"</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="p">}</span>
<span class="nv">after</span> <span class="s">"puppet_install"</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="p">}</span>
</code></pre>
</div>
<p>这几行就是 Rexfile 的任务定义主体格式。task 指令定义任务,任务会在具体的 -H 或者 -G 服务器上执行。其他都是可选项,desc内容会在 -Tv 的时候显示;before 和 after 定义的任务会在执行对应 task 之前或之后,在’'’rex命令执行处,即10.4.1.21本地’'’执行。</p>
<h2 id="section-1">常用指令介绍</h2>
<ul>
<li>run</li>
</ul>
<p>运行命令。如果有回调函数,那么会把 stdout 和 stderr 传给回调函数;如果没有,直接把 stdout 作为返回值。</p>
<p>比如:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">say</span> <span class="nv">run</span> <span class="s">"uptime"</span><span class="p">;</span>
<span class="nv">run</span> <span class="s">"nginx -v"</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span> <span class="k">my</span> <span class="p">(</span><span class="nv">$out</span><span class="p">,</span> <span class="nv">$err</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span> <span class="nv">say</span> <span class="nv">$err</span> <span class="p">};</span>
</code></pre>
</div>
<ul>
<li>file</li>
</ul>
<p>分发文件。语法类似 Puppet 的 file。支持 source、template、ensure、on_change 等操作。注意:rex 是顺序执行 Rexfile 的,所以不用设置 Puppet 的 require 指令。</p>
<p>比如:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">file</span> <span class="s">"/etc/yum.repos.d/xiaonei-private.repo"</span><span class="p">,</span>
<span class="nv">source</span> <span class="o">=></span> <span class="s">"repos/xiaonei-private.repo"</span><span class="p">;</span>
<span class="nv">file</span> <span class="s">"/etc/nginx/nginx.conf"</span><span class="p">,</span>
<span class="nv">content</span> <span class="o">=></span> <span class="nv">template</span><span class="p">(</span><span class="s">"templates/etc/nginx/nginx.conf.tpl"</span><span class="p">),</span>
<span class="nv">owner</span> <span class="o">=></span> <span class="s">"nginx"</span><span class="p">,</span>
<span class="nv">group</span> <span class="o">=></span> <span class="s">"nginx"</span><span class="p">,</span>
<span class="nv">mode</span> <span class="o">=></span> <span class="mi">644</span><span class="p">,</span>
<span class="nv">ensure</span> <span class="o">=></span> <span class="s">'file'</span><span class="p">,</span>
<span class="nv">on_change</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="nv">service</span> <span class="nv">nginx</span> <span class="o">=></span> <span class="s">"restart"</span><span class="p">;</span> <span class="p">};</span>
<span class="nv">file</span> <span class="s">"/etc/nginx/conf.d"</span><span class="p">,</span>
<span class="nv">ensure</span> <span class="o">=></span> <span class="s">"directory"</span><span class="p">,</span>
</code></pre>
</div>
<ul>
<li>pkg</li>
</ul>
<p>安装软件包,在早期版本命令写作 <code class="highlighter-rouge">install package => "nginx"</code> ,最近改成 <code class="highlighter-rouge">pkg</code> 了,更像 Puppet 语法了。</p>
<p>也支持传递数组作为 pkg 内容。另外,rex 还 提供了一个 update_package_db 指令,用于执行 <code class="highlighter-rouge">yum clean all</code> 或者 <code class="highlighter-rouge">apt-get update</code> 操作。这点是 Puppet 欠缺的。</p>
<p>比如:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">update_package_db</span><span class="p">();</span>
<span class="k">my</span> <span class="nv">$packages</span> <span class="o">=</span> <span class="k">case</span> <span class="nv">operating_system</span><span class="p">,</span>
<span class="nv">Debian</span> <span class="o">=></span> <span class="p">[</span><span class="s">"apache2"</span><span class="p">,</span> <span class="s">"libphp5-apache2"</span><span class="p">],</span>
<span class="nv">CentOS</span> <span class="o">=></span> <span class="p">[</span><span class="s">"httpd"</span><span class="p">,</span> <span class="s">"php5"</span><span class="p">],</span>
<span class="nv">pkg</span> <span class="nv">$packages</span><span class="p">,</span>
<span class="nv">ensure</span> <span class="o">=></span> <span class="s">"present"</span><span class="p">;</span>
</code></pre>
</div>
<p>ensure 也支持 present、absent、latest 等几种含义。同 Puppet。</p>
<ul>
<li>account</li>
</ul>
<p>用户管理原先用 <code class="highlighter-rouge">create_user</code> 和 <code class="highlighter-rouge">create_group</code> 指令,最近把 <code class="highlighter-rouge">create_user</code> 更新为 <code class="highlighter-rouge">account</code> 指令。</p>
<p>比如:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">create_group</span> <span class="s">'puppet'</span><span class="p">;</span>
<span class="nv">account</span> <span class="s">"puppet"</span><span class="p">,</span>
<span class="nv">ensure</span> <span class="o">=></span> <span class="s">"present"</span><span class="p">,</span>
<span class="nv">uid</span> <span class="o">=></span> <span class="mi">509</span><span class="p">,</span>
<span class="nv">home</span> <span class="o">=></span> <span class="s">'/home/puppet'</span><span class="p">,</span>
<span class="nv">comment</span> <span class="o">=></span> <span class="s">'Puppet Account'</span><span class="p">,</span>
<span class="nv">expire</span> <span class="o">=></span> <span class="s">'2015-05-30'</span><span class="p">,</span>
<span class="nv">groups</span> <span class="o">=></span> <span class="p">[</span><span class="s">'puppet'</span><span class="p">],</span>
<span class="nv">password</span> <span class="o">=></span> <span class="s">'puppet'</span><span class="p">,</span>
<span class="nb">system</span> <span class="o">=></span> <span class="mi">1</span><span class="p">,</span>
<span class="nv">no_create_home</span> <span class="o">=></span> <span class="nv">TRUE</span><span class="p">,</span>
<span class="nv">ssh_key</span> <span class="o">=></span> <span class="s">"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQChUw..."</span><span class="p">;</span>
</code></pre>
</div>
<ul>
<li>tail</li>
</ul>
<p>用来同时观测多台主机的日志的最新追加情况。应该是比较有用的一个小功能。代码如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">tail</span> <span class="s">"/var/log/syslog"</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$data</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$server</span> <span class="o">=</span> <span class="nv">Rex</span><span class="o">-></span><span class="nv">get_current_connection</span><span class="p">()</span><span class="o">-></span><span class="p">{</span><span class="s">'server'</span><span class="p">};</span>
<span class="k">print</span> <span class="s">"$server>> $data\n"</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<h2 id="section-2">远程主机详情相关变量</h2>
<p>Puppet 有专门的 Facts 变量来判定远程主机的详情。Rex 因为走 SSH 连接,不会在远程主机上跑一个 agent 来收集这些信息,所以还是通过远程执行命令的方式来提供相关内容。目前常用的几个函数(也可以认为是变量)有:</p>
<ul>
<li>is_redhat</li>
</ul>
<p>这个用来判断操作系统是否是 RedHat 系列。之前因为有一批 Debian 的机器,所以 Rexfile 里一直有这么个操作逻辑:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">if</span> <span class="p">(</span> <span class="nv">is_debian</span> <span class="p">)</span> <span class="p">{</span>
<span class="p">}</span> <span class="k">elsif</span> <span class="p">(</span> <span class="nv">is_redhat</span> <span class="p">)</span> <span class="p">{</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="p">}</span>
</code></pre>
</div>
<ul>
<li>operating_system_version</li>
</ul>
<p>这个用来判断具体的操作系统版本号。比如 CentOS5 跟 CentOS6 应该应用的操作就不一样,甚至 CentOS6.5 和 CentOS6.2 也有可能不一致。</p>
<p>比如 Rexfile 里的 1w10 任务:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">if</span> <span class="p">(</span> <span class="nv">is_redhat</span> <span class="ow">and</span> <span class="nv">operating_system_version</span> <span class="o">>=</span> <span class="mi">64</span> <span class="p">)</span>
<span class="p">}</span>
</code></pre>
</div>
<ul>
<li>route</li>
</ul>
<p>rex 可以收集的信息比 puppet 要多很多,比如网络相关、sysctl 相关等等。Rexfile 里的 1w10 任务用到了 route 信息来获取默认网关和网卡接口。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">my</span> <span class="p">(</span><span class="nv">$default_route</span><span class="p">)</span> <span class="o">=</span> <span class="nb">grep</span> <span class="p">{</span>
<span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="s">"flags"</span><span class="p">}</span> <span class="o">=~</span> <span class="sr">m/UG/</span> <span class="o">&&</span> <span class="p">(</span>
<span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="s">"destination"</span><span class="p">}</span> <span class="ow">eq</span> <span class="s">"0.0.0.0"</span> <span class="o">||</span>
<span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="s">"destination"</span><span class="p">}</span> <span class="ow">eq</span> <span class="s">"default"</span> <span class="p">)</span>
<span class="p">}</span> <span class="nv">route</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$default_route</span><span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$default_gw</span> <span class="o">=</span> <span class="nv">$default_route</span><span class="o">-></span><span class="p">{</span><span class="s">"gateway"</span><span class="p">};</span>
<span class="k">my</span> <span class="nv">$default_if</span> <span class="o">=</span> <span class="nv">$default_route</span><span class="o">-></span><span class="p">{</span><span class="s">"iface"</span><span class="p">};</span>
<span class="nv">run</span> <span class="s">"ip route change default via ${default_gw} dev ${default_if} initcwnd 10 initrwnd 10"</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<ul>
<li>connection</li>
</ul>
<p>在多台主机执行任务的时候,大多希望在输出的时候看到某条结果是哪个主机返回的。前面 tail 任务就用到了,不过写起来非常复杂的样子。其实 rex 提供给更简洁一点的写法。就是 connection->server。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">task</span> <span class="s">'tellmewhoyouare'</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">say</span> <span class="nv">connection</span><span class="o">-></span><span class="nv">server</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
<p>当前连接的服务器的整个信息,也可以通过 <code class="highlighter-rouge">get_system_information</code> 指令来获取,这两个指令其实是等同的。不过根据字面意思一般用来不同语境下。</p>
<p>这些信息如果要完整查看,可以通过 <code class="highlighter-rouge">dump_system_information</code> 指令来查看。这个命令跟 <code class="highlighter-rouge">print Dumper get_system_information()</code> 不一样的是,会把每个键作为单独变量。而这些变量就是可以直接用于 rex 的 template 里的内嵌变量。比如:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">listen</span> <span class="o"><</span><span class="nv">%</span><span class="err">=</span> <span class="err">$</span><span class="nv">eth0_ip</span> <span class="nv">%</span><span class="err">></span><span class="nv">:80</span><span class="p">;</span>
<span class="nv">visible_hostname</span> <span class="o"><</span><span class="nv">%</span><span class="err">=</span> <span class="err">$</span><span class="nv">hostname</span> <span class="nv">%</span><span class="err">></span>
</code></pre>
</div>
<p>不在 <code class="highlighter-rouge">dump_system_information</code> 清单里的变量,也想在 template 里使用的,就必须显式传递。这点和 Puppet 不一致,puppet 在 template 里可以通过 <code class="highlighter-rouge">scope.lookupvar()</code> 指令获取任意pp类里设定的变量,这一点完全无视词法作用域的存在==!</p>
<p>比如:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">file</span> <span class="s">'/etc/elasticsearch/elasticsearch.yml'</span><span class="p">,</span>
<span class="nv">content</span> <span class="o">=></span> <span class="nv">template</span><span class="p">(</span><span class="s">'files/es.yml.tmpl'</span><span class="p">,</span> <span class="nv">conf</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">clustername</span> <span class="o">=></span> <span class="s">'logstash'</span>
<span class="p">});</span>
</code></pre>
</div>
<p>对应的 es.yml.tmpl 里写作:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">clustername:</span> <span class="o"><</span><span class="nv">%</span><span class="err">=</span> <span class="err">$</span><span class="nv">conf</span><span class="o">-></span><span class="p">{</span><span class="s">'clustername'</span><span class="p">}</span> <span class="nv">%</span><span class="err">></span>
</code></pre>
</div>
<p>这样才行。</p>
PerlDancer 框架笔记
2014-06-12T00:00:00+08:00
perl
dancer
http://chenlinux.com/2014/06/12/perldancer-tips
<p>Dancer 是 Perl 的 web 开发框架,在 metacpan 上有 100 多个 like。其语法结构都起源自 Ruby 的 sinatra 框架,sinatra 曾经在自己官网上悬挂“perldancer is good”标语以示对 perldancer 的支持。Dancer 官网见: <a href="http://perldancer.org/">http://perldancer.org/</a> 本文系本人在部门 Wiki 上稍微写的几行介绍性质的笔记。</p>
<h2 id="section">简单示例</h2>
<p>Dancer 作为微框架,可以直接单文件快速运行简单的 web 功能。示例如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nv">Dancer</span><span class="p">;</span>
<span class="nv">get</span> <span class="s">'/'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">return</span> <span class="s">"hello world"</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">dance</span><span class="p">;</span>
</code></pre>
</div>
<p>然后直接通过 <code class="highlighter-rouge">perl test.pl</code> 命令既可以在 localhost:3000 运行起来一个 hello world 页面了。</p>
<h2 id="section-1">目录结构</h2>
<p>完整的 Dancer 应用,可以通过 <code class="highlighter-rouge">dancer -a MyApp</code> 命令创建,目录结构如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>MyApp/
├── bin
│ └── app.pl # 程序运行入口,可以直接通过./app.pl运行,也可以通过plackup -s Starman app.pl来切换其他高性能服务器
├── config.yml # 主配置文件
├── environments
│ ├── development.yml
│ └── production.yml
├── lib
│ └── MyApp.pm # Perl代码入口,route、controller、ORM 等都在 lib 下
├── Makefile.PL
├── MANIFEST
├── MANIFEST.SKIP
├── public # public/ 下的文件会直接作为静态文件发布,相当于 DocumentRoot
│ ├── 404.html
│ ├── 500.html
│ ├── css
│ │ ├── error.css
│ │ └── style.css
│ ├── dispatch.cgi
│ ├── dispatch.fcgi
│ ├── favicon.ico
│ ├── images
│ │ ├── perldancer-bg.jpg
│ │ └── perldancer.jpg
│ └── javascripts
│ └── jquery.js
├── t
│ ├── 001_base.t
│ └── 002_index_route.t
└── views # views/ 下的文件是页面模板,在 lib 里通过 template('index') 方式调用
├── index.tt
└── layouts
└── main.tt # layouts 是页面模板的底层模板,主底层模板可以在 config.yml 里指定
</code></pre>
</div>
<h2 id="section-2">常用插件</h2>
<p>目前用 Dancer 写的 CdnManage 平台,用到的插件包括:</p>
<ul>
<li>Dancer::Template::Xslate</li>
</ul>
<p>采用 Text::Xslate 作为模板引擎。xslate 引擎是用 XS 写的类 Perl6 语法模板引擎,性能很好。语法示例如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><: $object.accessor :>
<: $str :>
<: $array.0 :>
<: $hash.key :>
: for $arrayref -> $item {
index: <: $~item :> value: <: $item :>
: }
: if ( $var == nil ) {
: } else if ( $val == "text" ) {
: } else {
: while $dbh.fetch() -> $item {
: }
: }
</code></pre>
</div>
<p>注意,CdnManage 中,因为是从 TT2 模板迁移到 xslate 里的,所以单独配置了 config.yml,没有用 : 号而是沿用了 % 号。</p>
<ul>
<li>Dancer::Session::YAML</li>
</ul>
<p>采用 YAML 存储 session,这个作为内部应用足够了,升级的话应该用 mysql、mongo、elasticsearch之类的存储,都有现成插件。</p>
<ul>
<li>Plack::Middleware::Deflater</li>
<li>Plack::Middleware::ETag</li>
</ul>
<p>上面两个作为给 public/ 下文件加缓存和压缩的优化。在 config.yml 里添加如下配置即可使用:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="s">plack_middlewares</span><span class="pi">:</span>
<span class="pi">-</span>
<span class="pi">-</span> <span class="s">Plack::Middleware::Deflater</span>
<span class="pi">-</span> <span class="s">Plack::Middleware::ETag</span>
</code></pre>
</div>
<ul>
<li>Dancer::Plugin::Auth::Extensible</li>
</ul>
<p>给 route 加认证功能,有 require_role 和 require_user 两种形式,示例如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">get</span> <span class="s">'/admin'</span> <span class="o">=></span> <span class="nv">require_user</span> <span class="s">'admin'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{};</span>
<span class="nv">post</span> <span class="s">'/purge'</span> <span class="o">=></span> <span class="nv">require_role</span> <span class="sx">qr/^purge_\w+/</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{};</span>
</code></pre>
</div>
<ul>
<li>Dancer::Plugin::Email</li>
</ul>
<p>发邮件</p>
<ul>
<li>Dancer::Plugin::GearmanXS</li>
</ul>
<p>将需要较长时间运行完的任务通过 gearman 分发到其他后台任务脚本上去完成。</p>
<ul>
<li>Dancer::Plugin::Datebase</li>
</ul>
<p>数据库插件,可以直接按照 DBI 操作,也提供了简单的 quick_select/insert 等指令。示例如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">get</span> <span class="s">'/users/:id'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">template</span> <span class="s">'display_user'</span><span class="p">,</span> <span class="p">{</span>
<span class="nv">person</span> <span class="o">=></span> <span class="nv">database</span><span class="o">-></span><span class="nv">quick_select</span><span class="p">(</span><span class="s">'users'</span><span class="p">,</span> <span class="p">{</span> <span class="nv">id</span> <span class="o">=></span> <span class="nv">params</span><span class="o">-></span><span class="p">{</span><span class="nv">id</span><span class="p">}</span> <span class="p">}),</span>
<span class="p">};</span>
<span class="p">};</span>
</code></pre>
</div>
<p>如果在 config.yml 定义了多个库,则通过 <code class="highlighter-rouge">database('name')</code> 的方式来调用。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="s">Database</span><span class="pi">:</span>
<span class="s">connections</span><span class="pi">:</span>
<span class="s">puppet</span><span class="pi">:</span>
<span class="s">driver</span><span class="pi">:</span> <span class="s2">"</span><span class="s">SQLite"</span>
<span class="s">database</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/etc/puppet/webui/node_info.db"</span>
<span class="s">cdnmanage</span><span class="pi">:</span>
<span class="s">driver</span><span class="pi">:</span> <span class="s2">"</span><span class="s">mysql"</span>
<span class="s">database</span><span class="pi">:</span> <span class="s2">"</span><span class="s">cdnmanage"</span>
<span class="s">host</span><span class="pi">:</span> <span class="s2">"</span><span class="s">127.0.0.1"</span>
<span class="s">port</span><span class="pi">:</span> <span class="s">3306</span>
<span class="s">username</span><span class="pi">:</span> <span class="s2">"</span><span class="s">user"</span>
<span class="s">password</span><span class="pi">:</span> <span class="s2">"</span><span class="s">pass"</span>
<span class="s">connection_check_threshold</span><span class="pi">:</span> <span class="s">10</span>
<span class="s">on_connect_do</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">SET</span><span class="nv"> </span><span class="s">NAMES</span><span class="nv"> </span><span class="s">'utf8'"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">SET</span><span class="nv"> </span><span class="s">CHARACTER</span><span class="nv"> </span><span class="s">SET</span><span class="nv"> </span><span class="s">'utf8'"</span> <span class="pi">]</span>
</code></pre>
</div>
<p>更完善的 ORM 使用,见 Dancer::Plugin::DBIC 插件,他使用的是 DBIx::Class 框架做 ORM,示例如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">get</span> <span class="s">'/users/:user_id'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$user</span> <span class="o">=</span> <span class="nv">schema</span><span class="p">(</span><span class="s">'default'</span><span class="p">)</span><span class="o">-></span><span class="nv">resultset</span><span class="p">(</span><span class="s">'User'</span><span class="p">)</span><span class="o">-></span><span class="nv">find</span><span class="p">(</span><span class="nv">param</span> <span class="s">'user_id'</span><span class="p">);</span>
<span class="c1"># 如果只有一个默认的schema在config.yml里那么上面这行可以简写成下行</span>
<span class="nv">$user</span> <span class="o">=</span> <span class="nv">rset</span><span class="p">(</span><span class="s">'User'</span><span class="p">)</span><span class="o">-></span><span class="nv">find</span><span class="p">(</span><span class="nv">param</span> <span class="s">'user_id'</span><span class="p">);</span>
<span class="nv">template</span> <span class="nv">user_profile</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">user</span> <span class="o">=></span> <span class="nv">$user</span>
<span class="p">};</span>
<span class="p">};</span>
</code></pre>
</div>
<ul>
<li>Dancer::Plugin::ElasticSearch</li>
</ul>
<p>elasticsearch 插件,类似 Dancer::Plugin::Database;所以同理,也有更偏 ORM 一点的 Dancer::Plugin::ElasticModel 插件。</p>
<ul>
<li>Dancer::Plugin::Deferred</li>
</ul>
<p>页面消息提示插件。使用示例:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">hook</span> <span class="nv">before</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">request</span><span class="o">-></span><span class="nv">uri</span> <span class="o">=~</span> <span class="nv">m</span><span class="c1">#^/puppetdb/#</span>
<span class="ow">and</span> <span class="nv">request</span><span class="o">-></span><span class="nv">uri</span> <span class="o">!~</span> <span class="nv">m</span><span class="c1">#^/puppetdb/api/#</span>
<span class="ow">and</span> <span class="o">!</span><span class="nv">user_has_role</span><span class="p">(</span><span class="s">'SOM'</span><span class="p">)</span> <span class="p">)</span>
<span class="p">{</span>
<span class="nv">deferred</span> <span class="nv">error</span> <span class="o">=></span> <span class="s">'no permission'</span><span class="p">;</span>
<span class="nv">redirect</span> <span class="s">'/'</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">};</span>
</code></pre>
</div>
<p>然后在底层模板layouts/main.tt 中:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>%% if $deferred.error {
<div class="alert alert-success"> [% $deferred.error %] </div>
%% }
</code></pre>
</div>
<ul>
<li>Dancer::Plugin::Ajax</li>
</ul>
<p>扩展默认的 get/post/delete/put 指令,提供 ajax 指令。</p>
<ul>
<li>Dancer::Plugin::SimpleCRUD</li>
</ul>
<p>提供简便的数据库 CRUD 操作表单。目前 Puppet 的 SQLite 操作实例如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">simple_crud</span><span class="p">(</span>
<span class="nv">db_connection_name</span> <span class="o">=></span> <span class="s">'puppet'</span><span class="p">,</span>
<span class="nv">db_table</span> <span class="o">=></span> <span class="s">'node_info'</span><span class="p">,</span>
<span class="nv">key_column</span> <span class="o">=></span> <span class="s">'id'</span><span class="p">,</span>
<span class="nv">prefix</span> <span class="o">=></span> <span class="s">'node_info'</span><span class="p">,</span>
<span class="nv">record_title</span> <span class="o">=></span> <span class="s">'Puppet Node'</span><span class="p">,</span>
<span class="nv">deleteable</span> <span class="o">=></span> <span class="mi">1</span><span class="p">,</span>
<span class="nv">paginate</span> <span class="o">=></span> <span class="mi">50</span><span class="p">,</span>
<span class="nv">validation</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">classes</span> <span class="o">=></span> <span class="s">'/^(\w,?)+$/'</span><span class="p">,</span>
<span class="nv">role</span> <span class="o">=></span> <span class="s">'/^\w+$/'</span><span class="p">,</span>
<span class="nv">environment</span> <span class="o">=></span> <span class="s">'/^\w+$/'</span><span class="p">,</span>
<span class="p">},</span>
<span class="nv">message</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">classes</span> <span class="o">=></span> <span class="s">'enter like "puppetd,repos"'</span><span class="p">,</span>
<span class="nv">role</span> <span class="o">=></span> <span class="s">'an english word only'</span><span class="p">,</span>
<span class="p">},</span>
<span class="nv">display_columns</span> <span class="o">=></span> <span class="p">[</span><span class="sx">qw(node_fqdn environment role)</span><span class="p">],</span>
<span class="nv">custom_columns</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">include_classes</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">raw_column</span> <span class="o">=></span> <span class="s">'classes'</span><span class="p">,</span>
<span class="nv">transform</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">@classes</span> <span class="o">=</span> <span class="nb">split</span><span class="p">(</span> <span class="sr">/,/</span><span class="p">,</span> <span class="nb">shift</span> <span class="p">);</span>
<span class="k">my</span> <span class="nv">$self</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$role</span> <span class="o">=</span> <span class="nv">$self</span><span class="o">-></span><span class="p">{</span><span class="s">'role'</span><span class="p">};</span>
<span class="k">my</span> <span class="nv">$env</span> <span class="o">=</span> <span class="nv">$self</span><span class="o">-></span><span class="p">{</span><span class="s">'environment'</span><span class="p">};</span>
<span class="k">my</span> <span class="nv">@lines</span><span class="p">;</span>
<span class="nb">push</span> <span class="nv">@lines</span><span class="p">,</span> <span class="s">"<a href='/puppetdb/$env/$_/$role/view'>$_</a>"</span>
<span class="k">for</span> <span class="nv">@classes</span><span class="p">;</span>
<span class="k">return</span> <span class="nb">join</span><span class="p">(</span> <span class="s">" / "</span><span class="p">,</span> <span class="nv">@lines</span> <span class="p">);</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">);</span>
</code></pre>
</div>
Perl 编程的个人惯例
2014-06-12T00:00:00+08:00
perl
http://chenlinux.com/2014/06/12/perl-tips
<p>Perl 代码规范可以参考著名的《Perl 最佳实践》一书。当然,PBP 上的规定比较严格,实际生活中绝对多数 Perl 程序都无法通过以 PBP 规范编写的 Perl::Critic 模块的校验。本文仅为本人在部门 Wiki 上以部分常见用法作为示例的介绍性文档。</p>
<h2 id="section">格式化</h2>
<p>所有已经完成功能的 Perl 脚本,强烈推荐使用 Perl::Tidy 模块格式化其内容。具体命令为:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>perltidy your.pl && mv your.pl.tdy your.pl
</code></pre>
</div>
<h2 id="section-1">模板</h2>
<p>为调试和使用方便,强烈建议在所有 Perl 程序开始位置使用如下模板:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">utf8</span><span class="p">;</span>
</code></pre>
</div>
<p>这个模板最重要最常见的作用,就是说,程序内不允许直接使用未经初始化的变量,强制要求指定变量作用域范围,也不允许跨越词法作用域调用变量。</p>
<p>此外,考虑 CentOS6 已经成为我们线上主流操作系统,建议继续添加下行模板:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="mf">5.010</span><span class="p">;</span>
</code></pre>
</div>
<p>10 版本是 Perl5 的一次重大更新,添加了 state 变量、say 指令、// 判断符、%+ 正则捕获哈希、given-when流程和 ~~ 智能匹配符,都是比较常用和好用的功能。</p>
<h2 id="section-2">注释与文档</h2>
<p>Perl 注释以 ‘#’ 号开头,但是并没有提供方便的读取注释的方法。所以如果有需要,建议书写 POD 式的文档型注释。CPAN 提供有一系列模块处理程序内部的 POD 文档,比如可以直接从 POD 生成 –help 输出,README 文本等等。</p>
<p>POD 格式包括:</p>
<h3 id="section-3">标题</h3>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="o">=</span><span class="nv">pod</span>
<span class="err">标记文档开始</span>
<span class="o">=</span><span class="nv">head1</span> <span class="err">大标题</span>
<span class="err">标记为标题文档,类似</span> <span class="nv">HTML</span> <span class="err">的</span> <span class="sr"><h1></span> <span class="err">,同理还有</span> <span class="nv">head2</span><span class="sr">/3/</span><span class="mi">4</span>
<span class="o">=</span><span class="nv">over</span>
<span class="err">标记一段落开始</span>
<span class="o">=</span><span class="nv">item</span> <span class="err">元素</span>
<span class="err">标记该段落中某个列表元素</span>
<span class="o">=</span><span class="nv">back</span>
<span class="err">标记该段落结束。</span><span class="nv">over</span> <span class="err">和</span> <span class="nv">back</span> <span class="err">在用</span> <span class="nv">POD</span> <span class="err">书写函数注释的时候非常常见,每个函数上面一对</span>
</code></pre>
</div>
<h3 id="section-4">代码示例</h3>
<p>直接空四格,这点类似 markdown</p>
<h3 id="section-5">变量和链接格式的快捷书写方式</h3>
<p>C<code> 内含代码中如果本身带有<和>符号的,可以写作 C<< code >>的形式</和></p>
<p>L<name> 内含name为 CPAN 模块名,自动生成该模块在 CPAN 上的 url 地址连接</p>
<h2 id="modern-perl">modern perl</h2>
<h3 id="oop">OOP</h3>
<p>Perl5 采用 bless 指令将一个数据结构跟一个类名结合到一起就成为了类,其最简写法如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">package</span> <span class="nv">Foo</span> <span class="p">{</span> <span class="k">sub </span><span class="nf">new</span> <span class="p">{</span> <span class="nb">bless</span> <span class="nb">shift</span><span class="p">,</span> <span class="p">{}</span> <span class="p">}</span> <span class="p">}</span>
</code></pre>
</div>
<p>但是不推荐如此构建类。强烈推荐使用 Moo 模块完成 Perl5 的 OOP。文档见: https://metacpan.org/pod/Moo</p>
<p>最常用的属性、继承和角色三大功能示例如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">package</span> <span class="nv">Foo</span> <span class="p">{</span>
<span class="k">use</span> <span class="nv">Moo</span><span class="p">;</span>
<span class="p">}</span>
<span class="nb">package</span> <span class="nn">Bar::</span><span class="nv">Roles</span> <span class="p">{</span>
<span class="k">use</span> <span class="nn">Moo::</span><span class="nv">Role</span><span class="p">;</span>
<span class="nv">requires</span> <span class="s">'length'</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">width</span> <span class="p">{</span> <span class="k">return</span> <span class="s">'bar'</span> <span class="p">};</span>
<span class="p">}</span>
<span class="nb">package</span> <span class="nn">Foo::</span><span class="nv">Bar</span> <span class="p">{</span>
<span class="k">use</span> <span class="nv">Moo</span><span class="p">;</span>
<span class="nv">extends</span> <span class="s">'Foo'</span><span class="p">;</span>
<span class="nv">with</span> <span class="s">'Bar::Roles'</span><span class="p">;</span>
<span class="nv">has</span> <span class="nv">name</span> <span class="o">=></span> <span class="p">(</span> <span class="nv">is</span> <span class="o">=></span> <span class="s">'ro'</span><span class="p">,</span> <span class="nv">default</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="k">return</span> <span class="s">'foo'</span> <span class="p">}</span> <span class="p">);</span>
<span class="nv">has</span> <span class="nv">hight</span> <span class="o">=></span> <span class="p">(</span> <span class="nv">is</span> <span class="o">=></span> <span class="s">'lazy'</span> <span class="p">);</span>
<span class="k">sub </span><span class="nf">_build_hight</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$self</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">return</span> <span class="nv">$self</span><span class="o">-></span><span class="nv">name</span> <span class="o">.</span> <span class="nv">$self</span><span class="o">-></span><span class="nv">width</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">sub </span><span class="nf">length</span> <span class="p">{</span> <span class="k">return</span> <span class="nb">shift</span><span class="o">-></span><span class="nv">hight</span> <span class="p">};</span>
<span class="p">}</span>
<span class="k">my</span> <span class="nv">$fb</span> <span class="o">=</span> <span class="nn">Foo::</span><span class="nv">Bar</span><span class="o">-></span><span class="k">new</span><span class="p">(</span> <span class="nv">name</span> <span class="o">=></span> <span class="s">'myfoo'</span> <span class="p">);</span>
<span class="k">print</span> <span class="nv">$fb</span><span class="o">-></span><span class="nb">length</span><span class="p">;</span> <span class="c1"># myfoobar</span>
</code></pre>
</div>
<h3 id="todo">TODO</h3>
<p>Perl5 有独特的 TODO 语法叫 ‘…‘,在没有实现的地方,使用这个指令就可以了。不运行到这个地方就毫无影响,到这里就会直接显示“Unimplemented at line N”的返回。</p>
<p>示例如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">sub </span><span class="nf">somthing_todo</span> <span class="p">{</span>
<span class="o">...</span>
<span class="p">}</span>
</code></pre>
</div>
<h3 id="section-6">正则</h3>
<p>正则式是 Perl5 最强大和头疼的地方。这里不好说太多。只能说,能找到 CPAN 模块实现的,就不要自己写正则了。。。</p>
<p>如果要写,尽量使用 ‘/x’ 开启多行模式,然后每行写注释。</p>
<p>最常用的正则模块有 Regexp::Common 和 Regexp::Log。</p>
<p>日志处理方面,对 IP 归类 建议采用 Net::IP::Match::Trie 模块。此外,前缀树优化在 Perl5.14 开始成为正则引擎默认行为,所以请尽量使用新版本。</p>
<h3 id="section-7">文件操作</h3>
<p>open指令请使用三参数结构避免歧义以及恶意文件名问题:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">open</span> <span class="k">my</span> <span class="nv">$fh</span><span class="p">,</span> <span class="s">'>'</span><span class="p">,</span> <span class="s">'data.txt'</span> <span class="ow">or</span> <span class="nb">die</span> <span class="s">"$!"</span><span class="p">;</span>
</code></pre>
</div>
<p>在 5.10.1 以后,autodie 模块进入 corelist,所以可以这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nv">autodie</span><span class="p">;</span>
<span class="nb">open</span> <span class="k">my</span> <span class="nv">$fh</span><span class="p">,</span> <span class="s">'>'</span><span class="p">,</span> <span class="s">'data.txt'</span><span class="p">;</span>
</code></pre>
</div>
<p>更好的版本,推荐 Path::Tiny 模块,这是最近一年来在 metacpan 上多次周评分榜单第一的模块。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nn">Path::</span><span class="nv">Tiny</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$f</span> <span class="o">=</span> <span class="nv">path</span><span class="p">(</span><span class="s">'data.txt'</span><span class="p">);</span>
<span class="c1"># 不存在就先创建</span>
<span class="nv">$f</span><span class="o">-></span><span class="nv">touch</span> <span class="k">unless</span> <span class="nv">$f</span><span class="o">-></span><span class="nb">exists</span><span class="p">;</span>
<span class="c1"># 读取全部内容</span>
<span class="k">print</span> <span class="nv">$f</span><span class="o">-></span><span class="nv">slurp</span><span class="p">;</span>
<span class="c1"># 按行读取内容</span>
<span class="k">while</span> <span class="p">(</span><span class="nv">$f</span><span class="o">-></span><span class="nv">lines</span><span class="p">)</span> <span class="p">{</span> <span class="k">print</span> <span class="p">};</span>
<span class="c1"># 写入内容</span>
<span class="nv">$f</span><span class="o">-></span><span class="nv">spaw</span><span class="p">(</span><span class="s">'new data'</span><span class="p">);</span>
<span class="c1"># 追加内容</span>
<span class="nv">$f</span><span class="o">-></span><span class="nv">append</span><span class="p">(</span><span class="s">'newer data'</span><span class="p">);</span>
<span class="c1"># 目录操作</span>
<span class="k">my</span> <span class="nv">$d</span> <span class="o">=</span> <span class="nv">path</span><span class="p">(</span><span class="s">'/tmp'</span><span class="p">);</span>
<span class="k">for</span> <span class="p">(</span> <span class="nv">$d</span><span class="o">-></span><span class="nv">children</span><span class="p">(</span> <span class="sx">qr/^\.\w$/</span> <span class="p">)</span> <span class="p">)</span> <span class="p">{</span> <span class="k">print</span> <span class="nv">$_</span><span class="o">-></span><span class="nb">stat</span> <span class="p">};</span>
<span class="c1"># 类似 File::Find</span>
<span class="k">my</span> <span class="nv">$iter</span> <span class="o">=</span> <span class="nv">$d</span><span class="o">-></span><span class="nv">iterator</span><span class="p">({</span><span class="nv">recurse</span> <span class="o">=></span> <span class="mi">1</span><span class="p">});</span>
<span class="k">while</span> <span class="p">(</span> <span class="k">my</span> <span class="nv">$next</span> <span class="o">=</span> <span class="nv">$iter</span><span class="o">-></span><span class="p">()</span> <span class="p">)</span> <span class="p">{</span> <span class="k">print</span> <span class="nv">$_</span><span class="o">-></span><span class="nv">stringify</span> <span class="p">}</span>
</code></pre>
</div>
<p>而 File::Find 的 更好的替代版本,推荐 Path::Iterator::Rule 模块,速度也比上面 Path::Tiny 里的 ‘$d->iterator()’ 要好。</p>
<h3 id="section-8">网络操作</h3>
<p>HTTP 客户端一直以来一般使用 LWP::UserAgent 模块,不过作为小规模应用,推荐使用 HTTP::Tiny 模块,因为该模块已经在 Perl5.14 版本进入 corelist,在简单请求下性能也比 LWP 要好,不少模块已经在迁移依赖到 HTTP::Tiny 上。</p>
<p>而对于高性能需求,推荐使用 AnyEvent::HTTP 模块,基于 EV 事件驱动库,示例如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nn">AnyEvent::</span><span class="nv">HTTP</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">AnyEvent</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$cv</span> <span class="o">=</span> <span class="nv">AnyEvent</span><span class="o">-></span><span class="nv">condvar</span><span class="p">;</span>
<span class="k">for</span> <span class="k">my</span> <span class="nv">$url</span> <span class="p">(</span> <span class="nv">@urls</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$cv</span><span class="o">-></span><span class="nv">begin</span><span class="p">;</span>
<span class="nv">http_get</span> <span class="nv">$url</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$data</span><span class="p">,</span> <span class="nv">$headers</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="nv">$cv</span><span class="o">-></span><span class="nv">end</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nv">$cv</span><span class="o">-></span><span class="nb">recv</span><span class="p">;</span>
</code></pre>
</div>
<p>如需并发控制,事件流程的同步控制等功能,推荐使用 Promises 或者 Future 模块。同名的相关概念目前在 JS 和 Scala 中都有。</p>
<p>对于 HTML 解析,较为规范的情况下,不要再使用正则解析,而通过 DOM 树本身来做。以 XPath 路径查询的,推荐 Web::Scraper 模块;以 CSS 选择器查询的,推荐 Mojo::UserAgent 配合 Mojo::DOM 模块完成。示例如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">say</span> <span class="nn">Mojo::</span><span class="nv">UserAgent</span><span class="o">-></span><span class="k">new</span><span class="o">-></span><span class="nv">get</span><span class="p">(</span><span class="s">'www.perl.org'</span><span class="p">)</span><span class="o">-></span><span class="nv">res</span><span class="o">-></span><span class="nv">dom</span><span class="o">-></span><span class="nv">html</span><span class="o">-></span><span class="nv">head</span><span class="o">-></span><span class="nv">title</span><span class="o">-></span><span class="nv">text</span><span class="p">;</span>
</code></pre>
</div>
<p>非 HTTP 的网络编程,一般使用 IO::Socket::INET 模块,这里推荐继续使用 AnyEvent::Socket 模块,以利用 AnyEvent 的事件驱动性能。示例如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">tcp_server</span> <span class="nb">undef</span><span class="p">,</span> <span class="mi">8888</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span> <span class="k">my</span> <span class="p">(</span><span class="nv">$fh</span><span class="p">,</span> <span class="nv">$host</span><span class="p">,</span> <span class="nv">$port</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span> <span class="nb">syswrite</span> <span class="nv">$fh</span> <span class="s">"hello"</span><span class="p">;</span> <span class="p">}</span>
<span class="nv">tcp_connect</span> <span class="s">'localhost'</span><span class="p">,</span> <span class="mi">8888</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span> <span class="k">my</span> <span class="nv">$fh</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span> <span class="nb">sysread</span> <span class="nv">$fh</span><span class="p">,</span> <span class="k">my</span> <span class="nv">$msg</span><span class="p">,</span> <span class="mi">8</span><span class="p">;</span> <span class="k">print</span> <span class="nv">$msg</span><span class="p">;</span> <span class="p">}</span>
</code></pre>
</div>
<h3 id="web-">web 编程</h3>
<p>CGI.pm 已经从 Perl5.20 开始准备移出 corelist,所以不要再使用 CGI 做 web 编程了,Plack/PSGI 才是王道。作为简单应用,推荐使用 Dancer 微框架,完整的复杂应用,可以使用 Mojolicious 框架。</p>
<p>Dancer 框架示例如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nv">Dancer</span><span class="p">;</span>
<span class="nv">get</span> <span class="s">'/:name'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">return</span> <span class="s">'hello '</span><span class="o">.</span><span class="nv">param</span><span class="p">(</span><span class="s">'name'</span><span class="p">);</span>
<span class="p">};</span>
<span class="nv">dance</span><span class="p">;</span>
</code></pre>
</div>
用 LEK 组合处理 Nginx 访问日志
2014-06-11T00:00:00+08:00
logstash
elasticsearch
perl
syslog
anyevent
http://chenlinux.com/2014/06/11/nginx-access-log-to-elasticsearch
<p>Tengine 支持通过 syslog 方式发送日志(现在 Nginx 官方也支持了),所以可以通过 syslog 发送访问日志到 logstash 平台上,这种做法相对来说对线上服务器影响最小。最近折腾这件事情,一路碰到几个难点,把解决和优化思路记录一下。</p>
<h2 id="grok">少用 Grok</h2>
<p>感谢群里 @wood 童鞋提供的信息,Grok 在高压力情况下确实比较容易率先成为瓶颈。所以在日志格式可控的情况下,最好可以想办法跳过使用 Grok 的环节。在早先的 cookbook 里,就有通过自定义 LogFormat 成 JSON 样式的做法。我前年博客上也写过 nginx 上如此做的示例:<a href="http://chenlinux.com/2012/09/21/json-event-for-logstash/index.html">http://chenlinux.com/2012/09/21/json-event-for-logstash/index.html</a>。</p>
<p>不过这次并没有采用这种方式,而是定义日志格式成下面的样子,因为这种分割线方式对 Hive 平台同样是友好的。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>log_format syslog '$remote_addr|$host|$request_uri|$status|$request_time|$body_bytes_sent|'
'$upstream_addr|$upstream_status|$upstream_response_time|'
'$http_referrer|$http_add_x_forwarded_for|$http_user_agent';
access_log syslog:user:info:10.4.16.68:29125:tengine syslog ratio=0.1;
</code></pre>
</div>
<p>那么不用 Grok 怎么做呢?这里有一个很炫酷的写法。下面是 logstash 配置里 filter 段的实例:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>filter {
ruby {
remove_field => ['@version', 'priority', 'timestamp', 'logsource', 'severity', 'severity_label', 'facility', 'facility_label', 'pid','message']
init => "@kname = ['client','servername','url','status','time','size','upstream','upstreamstatus','upstreamtime','referer','xff','useragent']"
code => "event.append(Hash[@kname.zip(event['message'].split('|'))])"
}
mutate {
convert => ["size", "integer", "time", "float", "upstreamtime", "float"]
}
geoip {
source => "client"
fields => ["country_name", "region_name", "city_name", "real_region_name", "latitude", "longitude"]
remove_field => [ "[geoip][longitude]", "[geoip][latitude]" ]
}
}
</code></pre>
</div>
<p>而要达到跟这段 ruby+mutate 效果一致的 grok ,写法是这样的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>filter {
grok {
match => ["message", "%{IPORHOST:client}\|%{HOST:servername}\|%{URIPATHPARAM:url}\|%{NUMBER:status}\|(?:%{NUMBER:time:int}|-)\|(?:%{NUMBER:size}|-)\|(?:%{HOSTPORT:upstream}|-)\|(?:%{NUMBER:upstreamstatus}|-)\|(?:%{NUMBER:upstreamtime:int}|-)\|(?:%{URI:referer}|-)\|%{GREEDYDATA:xff}\|%{GREEDYDATA:useragent}"]
remove_field => ['@version', 'priority', 'timestamp', 'logsource', 'severity', 'severity_label', 'facility', 'facility_label', 'pid','message']
}
}
</code></pre>
</div>
<h1 id="syslog-">syslog 瓶颈</h1>
<p>运行起来以后,通过 Kibana 看到的全网 tengine 带宽只有 60 MBps左右,这个结果跟通过 NgxAccounting 统计输出的结果差距太大了。明显是有问题。</p>
<p>首先怀疑不会是 nginx.conf 通过 Puppet 下发重启的时候有问题吧?实际当然没有。</p>
<p>这时候运行 <code class="highlighter-rouge">netstat -pln | grep 29125</code> 命令,发现 <code class="highlighter-rouge">Recv-Q</code> 已经达到了 228096,并且一致维持在这个数没有变化。</p>
<p>由于之前对 ES 写入速度没太大信心,所以这时候的反应就是去查看 ES 服务器的状态,结果其实服务器 idle% 在 80% 以上,各种空闲,Kibana 上搜索反应也非常快。通过 top 命令看具体的线程情况,logstash 的 output/elasticsearch worker 本身占用资源就很少。包括后来实际也尝试了加大 output 的 workers 数量,加大 bin/logstash -w 的 filter worker 数量,其实都没用。</p>
<p>那么只能是 input/syslog 就没能收进来了。</p>
<p>之前写 filter 的时候,开过 -vv 模式,所以注意到过 input/syslog 里是利用 Logstash::Filter::Grok 来判定切割 syslog 内容的。按照前一节的说法,那确实可能是在收 syslog 的时候性能跟不上啊?</p>
<p>于是去翻了一下 Logstash::Input::Syslog 的代码,主体逻辑很简单,就是 <code class="highlighter-rouge">Thread.new { UDPSocket.new }</code> 这样。也就是说是一个单线程监听 UDP 端口!</p>
<p>然后我又下载了同为 Ruby 写的日志收集框架 fluentd 的 syslog 插件看看源代码,fluent-plugin-syslog 里,用的是 Cool.io 库作 UDP 异步处理。好吧,其实在此之前我只知道 EventMachine 库。。。不过由于 Logstash 是 JRuby 平台,又不清楚其 event 代码(以前基本只是看各种 plugin 的代码就够了),担心这么把 em 加上去会不会不太好。所以在摸清 logstash 代码之前,先用自己最熟悉的手段,搞定这个问题:</p>
<p><strong>用 Perl 的高性能 EV 库解决</strong></p>
<p>前年我同样提到过 Perl 也有仿照 Logstash 写的框架叫 Message::Passing,这个框架就是用 AnyEvent 和 Moo 写的,性能绝对没问题。不过各种插件和文档比较潦草,要想兼容现在 logstash 1.4 的 schema 比较费劲。所以,最后我选择了自己根据 tengine 日志的情况单独写一个脚本,结果如下:</p>
<script src="https://gist.github.com/chenryn/7c922ac424324ee0d695.js"></script>
<p>80 行左右的代码,从 input 到 output 都是 anyevent 驱动。( Search::Elasticsearch::Async 默认是基于 AnyEvent::HTTP 的,不过用 Promises 模块做了封装,所以写起来好像看不太出来~)</p>
<p>最终到 elasticsearch 里的数据结构跟 logstash 一模一样,之前配置好的 Kibana 样式完全不需要变动。而实际运行起来以后,Recv-Q 虽然不是一直保持在 0,但是偶然累积的队列也肯定会在几秒钟内被读取处理完毕。完全达到了效果。Kibana 上,带宽图回复到了跟 NgxAccounting 统计结果一样的 300 MBps 。成功!</p>
<p><img src="/images/uploads/ngx-syslog-flow-diff.png" alt="" /></p>
配合 avbot 的 HTTP 接口做自动应答的 Perl 脚本
2014-06-08T00:00:00+08:00
perl
anyevent
devops
http://chenlinux.com/2014/06/08/perl-script-for-avbot
<p>前两天<a href="http://chenlinux.com/2014/06/04/record-webqq-logs-by-avbot">博客里介绍了 avbot</a>,其中提到 avbot 提供了 HTTP 接口可以收发信息。那么,我们就可以自己写脚本来实现比原先的 <code class="highlighter-rouge">.qqbot help</code> 更详细的自动应答啦。今晚有空就写了几行 Perl ,实现了一个简单的扩展:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nv">utf8</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">JSON::</span><span class="nv">XS</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">AnyEvent</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">AnyEvent::</span><span class="nv">HTTP</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$f</span> <span class="o">=</span> <span class="p">{</span>
<span class="nv">help</span> <span class="o">=></span> <span class="s">".logstashbot support subcommand:\n\t"</span><span class="p">,</span>
<span class="nv">grok</span> <span class="o">=></span> <span class="s">'请主动使用 http://grokdebug.herokuapp.com'</span><span class="p">,</span>
<span class="nv">tnnd</span> <span class="o">=></span> <span class="s">'请直接说问题不要浪费口水问有人帮忙么'</span><span class="p">,</span>
<span class="nv">book</span> <span class="o">=></span> <span class="s">'支持原作者,请购买 www.logstashbook.com 上电子版'</span><span class="p">,</span>
<span class="p">};</span>
<span class="nv">$f</span><span class="o">-></span><span class="p">{</span><span class="s">'help'</span><span class="p">}</span> <span class="o">.=</span> <span class="nb">join</span><span class="p">(</span><span class="s">"\n\t"</span><span class="p">,</span> <span class="nb">keys</span> <span class="nv">%$f</span><span class="p">);</span>
<span class="nv">$</span><span class="nn">AnyEvent::HTTP::</span><span class="nv">TIMEOUT</span> <span class="o">=</span> <span class="mi">86400</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$url</span> <span class="o">=</span> <span class="s">'http://127.0.0.1:6176/message'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$cv</span> <span class="o">=</span> <span class="nv">AnyEvent</span><span class="o">-></span><span class="nv">condvar</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$ua</span><span class="p">;</span><span class="nv">$ua</span> <span class="o">=</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">$cv</span><span class="o">-></span><span class="nv">begin</span><span class="p">;</span>
<span class="nv">http_get</span> <span class="nv">$url</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$data</span><span class="p">,</span> <span class="nv">$header</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$hash</span> <span class="o">=</span> <span class="nv">decode_json</span> <span class="nv">$data</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$msg</span> <span class="o">=</span> <span class="nv">$hash</span><span class="o">-></span><span class="p">{</span><span class="s">'message'</span><span class="p">}{</span><span class="s">'text'</span><span class="p">};</span>
<span class="k">my</span> <span class="nv">$from</span> <span class="o">=</span> <span class="s">'@'</span> <span class="o">.</span> <span class="nv">$hash</span><span class="o">-></span><span class="p">{</span><span class="s">'who'</span><span class="p">}{</span><span class="s">'nick'</span><span class="p">}</span> <span class="o">.</span> <span class="s">'('</span> <span class="o">.</span> <span class="nv">$hash</span><span class="o">-></span><span class="p">{</span><span class="s">'who'</span><span class="p">}{</span><span class="s">'code'</span><span class="p">}</span> <span class="o">.</span> <span class="s">")\n"</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$msg</span> <span class="o">=~</span> <span class="sr">/^\.logstashbot (\w+)/</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$body</span> <span class="o">=</span> <span class="nv">encode_json</span><span class="p">({</span>
<span class="nv">protocol</span> <span class="o">=></span> <span class="nb">delete</span> <span class="nv">$hash</span><span class="o">-></span><span class="p">{</span><span class="s">'protocol'</span><span class="p">},</span>
<span class="nv">channel</span> <span class="o">=></span> <span class="nb">delete</span> <span class="nv">$hash</span><span class="o">-></span><span class="p">{</span><span class="s">'channel'</span><span class="p">},</span>
<span class="nv">message</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">text</span> <span class="o">=></span> <span class="nv">$from</span> <span class="o">.</span> <span class="p">(</span> <span class="nv">$f</span><span class="o">-></span><span class="p">{</span><span class="nv">$1</span><span class="p">}</span> <span class="sr">//</span> <span class="nv">$f</span><span class="o">-></span><span class="p">{</span><span class="s">'help'</span><span class="p">}</span> <span class="p">),</span>
<span class="p">},</span>
<span class="p">});</span>
<span class="nv">$cv</span><span class="o">-></span><span class="nv">begin</span><span class="p">;</span>
<span class="nv">http_post</span> <span class="nv">$url</span><span class="p">,</span> <span class="nv">$body</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">$cv</span><span class="o">-></span><span class="nv">end</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="nv">$ua</span><span class="o">-></span><span class="p">();</span>
<span class="nv">$cv</span><span class="o">-></span><span class="nv">end</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="nv">$ua</span><span class="o">-></span><span class="p">();</span>
<span class="nv">$cv</span><span class="o">-></span><span class="nb">recv</span><span class="p">;</span>
</code></pre>
</div>
<p>原先是打算在回调里 <code class="highlighter-rouge">undef $ua</code> 然后通过 <code class="highlighter-rouge">AnyEvent->timer</code> 里检测 $ua 是否还在,否则再起来的方式。后来一想 <code class="highlighter-rouge">timer</code> 还有间隔,直接函数内部通过 <code class="highlighter-rouge">$cv->end</code> 控制计数,不断的重新运行 <code class="highlighter-rouge">$ua->()</code> 来保持持续获取,间隔更短,就改成现在这样了。</p>
用 Perl5 改写 skyline 异常检测算法
2014-06-04T00:00:00+08:00
monitor
python
numpy
PDL
perl
skyline
http://chenlinux.com/2014/06/04/skyline-port-to-perl
<p>一直以来都知道 Perl5 里也有类似 numpy 的库叫 PDL,但是因为上手资料比较少,官网文档比较烂,就没认真看过。这次因为要了解 skyline 里用到的 9 种异常检测算法的具体原理,正好一一对照重写一下,当做是学习 PDL 了。</p>
<p>最终修改完的 Perl5 版如下:</p>
<script src="https://gist.github.com/chenryn/43315b6c7ddaf9c39aab.js"></script>
<p>~~要承认 PDL 在上手方面比不过 numpy,比如取数组长度,PDL 里居然写作 <code class="highlighter-rouge">$p->nelem</code>;取数组最后一个元素的值,更是要写作 <code class="highlighter-rouge">$p->index($p->nelem - 1)</code> 这么长!相比在 numpy 方面几乎看起来还是跟操作原生的 python 类型一样。。妈蛋 PDL 你多重载几个操作符会死啊!~~</p>
<p><strong>2014 年 06 月 09 日更新:在blogs.perl.org上得到指点,可以用 <code class="highlighter-rouge">$p->at(-1)</code> 来获取。PDL 自己的文档里 <code class="highlighter-rouge">->at()</code> 的示例都是获取数组的……</strong></p>
<p>稍微复杂一点的多维操作 PDL 还是很方便的。比如程序里 <code class="highlighter-rouge">least_squares</code> 检验法的时候,numpy 有这么一句:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">A</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">vstack</span><span class="p">([</span><span class="n">x</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">))])</span><span class="o">.</span><span class="n">T</span>
</code></pre>
</div>
<p>而在 PDL 里可以写作:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">my</span> <span class="nv">$A</span> <span class="o">=</span> <span class="nv">$x</span><span class="o">-></span><span class="nv">dummy</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">-></span><span class="nv">append</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
</code></pre>
</div>
<p>PDL 里也有 ones() 函数来生成全部由 1 构成的数组,不过我觉得上面这个写法明显更好理解最终目的,就是90°倒转数组然后每个元素作为子数组后面加第二个元素嘛。</p>
<p><em>当然,比较好玩的是最后我发现 <code class="highlighter-rouge">least_squares</code> 在 PDL 里可以直接搞出来结果,不用这么复杂</em></p>
<p>比较基础的数值统计还是比较好搞的,麻烦的是一些现成的正态分布检验法。python 版里使用的是 <a href="http://en.wikipedia.org/wiki/Kolmogorov-Smirnov_test">K-S 检验法</a>——其实只是命名,里面实际还用了 <a href="http://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test">A-D 检验法</a>做改进——我还记得这是 skyline 开源以后社区人帮忙实现的,Etsy 一开始都没有。按说 K-S 检验法是非常基础的一个,但是我找遍了 CPAN 确实就没有(大概是因为 Perl 里调用 R 太方便了,大家都习惯直接用 <a href="https:://metacpan.org/pod/Statistics::R">Statistics::R</a> 模块吧)。于是最后这个改成 <a href="http://en.wikipedia.org/wiki/Shapiro%E2%80%93Wilk_test">S-W 检验法</a>。</p>
<p><strong>根据 SPSS 的规范,一般在数值序列长度小于 5000 的时候,S-W 检验法可信度高于 K-S 检验法;大于 5000 的时候,K-S 检验法可信度大于 S-W 检验法。</strong></p>
<p>考虑这里一般只会检查最近一个小时的数据。一个小时内就算一秒钟一次也就是 3600 个点。事实上应该至少是 10 秒钟出一个统计值才会做比较。那么也就是几百个点,用 S-W 检验法应该更有效。</p>
<p>在重写这个脚本的时候,找到了很多关于这方面的资料,下面这两个链接应该是非常不错:</p>
<ol>
<li><a href="http://www.itl.nist.gov/div898/handbook/index.htm">http://www.itl.nist.gov/div898/handbook/index.htm</a></li>
<li><a href="http://www.perlmonks.org/?node=Stats%3A%20Testing%20whether%20data%20is%20normally%20(Gaussian)%20distributed">http://www.perlmonks.org/?node=Stats%3A%20Testing%20whether%20data%20is%20normally%20(Gaussian)%20distributed</a></li>
</ol>
<p>此外,脚本中本身用到的 <a href="http://www.ta-lib.org">ta-lib</a> 和 <a href="https:://metacpan.org/pod/Statistics::Distributions">Statistics::Distributions</a> 模块也还有更多的算法函数提供,值得留意。</p>
<p>注:PDL::Finance::Talib 模块必须先自己编译了 ta-lib 依赖后才能安装。之前测试在美团云主机上做的,结果还安装失败。后来发现是内存不够大==!然后在作者的指导下学会一招,在内存不够大的机器上,可以删除掉 CCFLAGS 里的 <code class="highlighter-rouge">-pipe</code> 参数,也能正常编译通过。</p>
用 avbot 机器人记录 QQ 群聊天记录
2014-06-04T00:00:00+08:00
devops
http://chenlinux.com/2014/06/04/record-webqq-logs-by-avbot
<p>这是一件蛮有趣的事情。我因为做 logstash 的 QQ 群管理员,碰到了一个幸福的烦恼:群里有不少高水平且乐于分享的朋友时常给人解答问题,而且一来一回的能牵扯出来不少让人眼前一亮的实践,但是 QQ 聊天记录不像邮件列表和 IRC 那样可以很方便的长期保存共享给后来人学习查找!这简直是国内参与开源技术最头疼的一件事情了,知识没法复用,偏偏越是需要这些知识的人,越是喜欢通过 QQ 来寻求帮助!前两天偶然想到,其实可以通过机器人潜水进来获取聊天记录,然后发布出来!询问了一下 <a href="http://weibo.com/biergaizi">@比尔盖子V</a> 童鞋,他推荐给我 <a href="http://wiki.avplayer.org/Avbot">avbot</a> 项目。#妈蛋这名字怎能不吐槽#</p>
<p>作者非常 nice 的提供好了 RPM 可以直接安装在服务器上。所以安装步骤真的就没啥可讲的了。</p>
<p>不过这个项目本意是做 QQ、IRC 和 XMPP 的互联互通,所以把心思用来了 <code class="highlighter-rouge">--map</code> 的实现,作为我们这里只想单单记录 QQ 群聊天记录来说,它不支持指定只获取某个群的记录,所以最好的办法就是新申请一个 QQ 号,只加这一个群……</p>
<p>运行起来以后,会在当前目录下生成一个 <code class="highlighter-rouge">avlog.db</code> 库,记录聊天记录,同时生成一个 QQ 群号命名的目录,里面按日期存放当天的聊天记录的 HTML 文件。直接用 nginx 发布出来就好啦!</p>
<p>照搬 avbot 官网 demo 页面做好了 logstash 群聊天记录的查看搜索页,见:<a href="http://logstash.chenlinux.com/">http://logstash.chenlinux.com/</a></p>
<p>下一步可以做的事情是做自动应答。已经测试过可以通过 RPC 接口收发消息。不过昨天碰到的一个怪事情是,没能准确收到 QQ 群号,于是变成了 none,结果发送就一直失败。这个重启进程让他重新获得一次就可以了。</p>
<p>收消息示例:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>curl 'http://localhost:6176/message'
{
"protocol": "qq",
"channel": "315428175",
"room":
{
"code": "3614128622",
"groupnumber": "315428175",
"name": "Logstash"
},
"op": "0",
"who":
{
"code": "225519360",
"nick": "田间",
"name": "田间",
"qqnumber": "",
"card": ""
},
"preamble": "qq(田间): ",
"message":
{
"text": "我们这暂时没运维 "
}
}
</code></pre>
</div>
<p>发消息示例:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>curl -XPOST http://localhost:6176/message -d '{"protocol":"qq","channel":"315428175","message":{"text":"Hi, my name is logstashbot, this message came from curl command!"}}'
</code></pre>
</div>
直接从 elasticsearch 获取数据进入 skyline 异常检测
2014-06-04T00:00:00+08:00
monitor
python
elasticsearch
skyline
http://chenlinux.com/2014/06/04/elasticsearch-direct-to-skyline
<p>这几天搭建 elasticsearch 集群做日志分析,终于有机会可以实际跑一下 skyline 的效果。不过比较麻烦的事情是,skyline 是一个比较完备的系统而不是插件,要求我们把数据通过 msgpack 发过去存到 redis 里。这是个很没有道理的做法,早在去年刚看到这个项目的时候我就在博客里写下了愿景是应该用 elasticsearch 替换掉 redis。等了这么久没等到,干脆就自己动手实现。修改后,skyline 其余的程序完全可以直接扔掉,只留下这一个脚本定时运行就够了:</p>
<script src="https://gist.github.com/chenryn/309bed093f6a7084c855.js"></script>
<p>其实改动的地方很少~这让我愈发不理解 etsy 原来那样做的理由了。</p>
<p>这里面主要就是拼了一下 elasticsearch 的 <code class="highlighter-rouge">date_histogram</code> 类型的 facet 请求,获取最近 1 个小时的每 5 分钟统计值构成的时间序列数据。然后发给前面那些检验算法。</p>
<p>之前用过 js 和 perl 的 elasticsearch 客户端,对象封装的都蛮细的,而 python 的这个客户端写起来就非常像 curl 命令了。</p>
<p>如果要推广用,把里面这个 <code class="highlighter-rouge">code.504</code> 提出来做一个可配置项就行了。</p>
巧用 Puppet 的 stdlib 库
2014-05-28T00:00:00+08:00
devops
puppet
ruby
http://chenlinux.com/2014/05/28/stdlib-of-puppet
<p>这几天上线机器给 Elasticsearch 集群扩容,开始撰写 Puppet 的 elasticsearch 类来规范化管理。这里碰到一个小问题,相信在很多大容量集群的机器上都会有。那就是每台机器上都挂载有十几二十块磁盘,怎么用 Puppet 给快速方便的创建各磁盘上的工作目录呢?</p>
<p>一个一个写 File 资源申明肯定不可取;File 资源申明支持接受数组,但是二十多个元素写一个大数组也没方便到哪里去。有没有比较简单的办法来生成这个大数组,而不是手写呢?</p>
<p>有,就是使用 Puppet 官方出的这个 stdlib 库 <a href="http://forge.puppetlabs.com/puppetlabs/stdlib">http://forge.puppetlabs.com/puppetlabs/stdlib</a>。</p>
<p>安装方法很简单,在 Puppet Master 上运行命令 <code class="highlighter-rouge">puppet module install puppetlabs-stdlib</code> 即可。</p>
<p>因为 puppet 默认会分发所有 module 的 lib/ 目录,所以即便你没有在自己的类里 <code class="highlighter-rouge">import stdlib</code>,也是可以直接使用它提供的各种函数的。</p>
<p>下面就是我的 elsticsearch 类配置:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="n">elasticsearch</span> <span class="p">{</span>
<span class="vg">$esdatadir</span> <span class="o">=</span> <span class="n">suffix</span><span class="p">(</span> <span class="n">prefix</span><span class="p">(</span> <span class="n">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="vg">$:</span><span class="ss">:datadircount</span><span class="o">-</span><span class="mi">1</span><span class="p">),</span> <span class="s1">'/data'</span><span class="p">),</span> <span class="s1">'/elasticsearch'</span><span class="p">)</span>
<span class="n">package</span> <span class="p">{[</span><span class="s1">'java-1.7.0-openjdk'</span><span class="p">,</span> <span class="s1">'elasticsearch'</span><span class="p">]:</span>
<span class="k">ensure</span> <span class="o">=></span> <span class="s1">'present'</span><span class="p">,</span>
<span class="nb">require</span> <span class="o">=></span> <span class="no">Class</span><span class="p">[</span><span class="s1">'repos'</span><span class="p">],</span>
<span class="p">}</span><span class="o">-></span>
<span class="n">file</span> <span class="p">{</span><span class="vg">$esdatadir</span><span class="p">:</span>
<span class="k">ensure</span> <span class="o">=></span> <span class="s1">'directory'</span><span class="p">,</span>
<span class="n">owner</span> <span class="o">=></span> <span class="s1">'elasticsearch'</span><span class="p">,</span>
<span class="p">}</span><span class="o">-></span>
<span class="n">file</span> <span class="p">{</span><span class="s1">'/etc/elasticsearch/elasticsearch.yml'</span><span class="p">:</span>
<span class="k">ensure</span> <span class="o">=></span> <span class="s1">'file'</span><span class="p">,</span>
<span class="n">owner</span> <span class="o">=></span> <span class="s1">'elasticsearch'</span><span class="p">,</span>
<span class="n">content</span> <span class="o">=></span> <span class="n">template</span><span class="p">(</span><span class="s1">'elasticsearch/elasticsearch.yml.erb'</span><span class="p">),</span>
<span class="p">}</span>
<span class="p">}</span><span class="o">~></span>
<span class="n">service</span> <span class="p">{</span><span class="s1">'elasticsearch'</span><span class="p">:</span>
<span class="k">ensure</span> <span class="o">=></span> <span class="kp">true</span><span class="p">,</span>
<span class="n">enable</span> <span class="o">=></span> <span class="kp">true</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>其中 <code class="highlighter-rouge">$::datadircount</code> 是我自定义的 Facts 变量,插件代码见两年前的博客<a href="http://chenlinux.com/2012/05/10/quick-start-for-puppet-facter-erb">《puppet安装/Facter插件和puppet模板编写》</a>。</p>
<p>然后 <code class="highlighter-rouge">elasticsearch.yml.erb</code> 里的数据目录配置定义如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">path</span><span class="p">.</span><span class="nf">data</span><span class="p">:</span>
<span class="o"><</span><span class="sx">% scope.lookupvar("elasticsearch::esdatadir").each </span><span class="k">do</span> <span class="o">|</span><span class="n">dir</span><span class="o">|</span> <span class="o">-</span><span class="sx">%>
- <%= dir %></span>
<span class="o"><</span><span class="sx">% end </span><span class="o">%></span>
</code></pre>
</div>
<p><code class="highlighter-rouge">puppetlibs-stdlib</code> 实现了很多对基础类型的扩展函数,比如本例中用到了 <code class="highlighter-rouge">range</code>、<code class="highlighter-rouge">prefix</code> 和 <code class="highlighter-rouge">suffix</code> 三个。依次生成了 1 到 N 的数组,给数组每个元素加上 <code class="highlighter-rouge">/data</code> 前缀字符串,再给每个元素加上 <code class="highlighter-rouge">/elasticsearch</code> 后缀字符串,最后变成了 <code class="highlighter-rouge">/dataN/elasticsearch</code> 这种格式的元素构成的数组。</p>
<p><code class="highlighter-rouge">puppetlibs-stdlib</code> 实现的非常漂亮的地方是,很多函数都根据常见用途提供了不同场景下的不同行为。</p>
<ul>
<li>比如 <code class="highlighter-rouge">range</code> 即可以 1 到 N,也可以 01 到 NN,甚至可以先加上 prefix 后再 ‘/data1’ 到 ‘/dataN’ 都支持。</li>
<li>比如 <code class="highlighter-rouge">unique</code> 既可以针对字符串去重,也可以针对数组元素去重。</li>
</ul>
<p>更多函数说明,见源码仓库 <a href="https://github.com/puppetlabs/puppetlabs-stdlib/blob/master/README.markdown">README</a> 文档。</p>
XS 初体验
2014-05-20T00:00:00+08:00
perl
c
xs
perl
http://chenlinux.com/2014/05/20/my-first-experience-of-perlxs
<p>今天翻 ganglia 源代码发现两年前加上了 <code class="highlighter-rouge">perl_module</code> 的<a href="http://t.cn/Rvwav9T">支持</a>,不过跟 <code class="highlighter-rouge">python_module</code> 相比,<code class="highlighter-rouge">descriptors</code> 里的 <code class="highlighter-rouge">call_back</code> 不是真的写作回调函数而是写作和实际函数同名的字符串,这点让我觉得很别扭和奇怪,于是想到去看看 gmond 里内嵌的 perl 解释程序是怎么做这步的。顺带就第一次动手写了一点 XS 代码,这里一并发上来,留作存档。</p>
<p>示例代码框架源自上周 Dancer 作者 SawyerX 发布的 <a href="https://github.com/xsawyerx/xs-fun">XS-Fun 项目</a>。所以这里如何使用 <code class="highlighter-rouge">h2xs</code> 命令创建 XS 模块文件就不讲解了。</p>
<p>主要分作五个小示例,由最简单到很简单依次如下:</p>
<h1 id="section">返回一个字符串</h1>
<p>编辑 XSFun.xs 内容如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="cp">#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "ppport.h"
</span>
<span class="cm">/* C functions */</span>
<span class="n">MODULE</span> <span class="o">=</span> <span class="n">XSFun</span> <span class="n">PACKAGE</span> <span class="o">=</span> <span class="n">XSFun</span>
<span class="cp"># XS code
</span>
<span class="n">SV</span> <span class="o">*</span>
<span class="n">runcb</span><span class="p">()</span>
<span class="n">CODE</span><span class="o">:</span>
<span class="n">STRLEN</span> <span class="n">len</span><span class="p">;</span>
<span class="k">const</span> <span class="kt">char</span><span class="o">*</span> <span class="n">str</span> <span class="o">=</span> <span class="s">"testsub"</span><span class="p">;</span>
<span class="n">SV</span><span class="o">*</span> <span class="n">val</span> <span class="o">=</span> <span class="n">newSVpv</span><span class="p">(</span><span class="n">str</span><span class="p">,</span> <span class="n">len</span><span class="p">);</span>
<span class="n">RETVAL</span> <span class="o">=</span> <span class="n">val</span><span class="p">;</span>
<span class="n">OUTPUT</span><span class="o">:</span> <span class="n">RETVAL</span>
</code></pre>
</div>
<p>这个其实就相当于 <code class="highlighter-rouge">sub runcb { return "testsub" }</code> 。</p>
<h1 id="section-1">返回一个哈希的指定键的值</h1>
<p>因为起因是 gmond 里的代码,所以这里就开始主要研究如何解析 descriptor 哈希的键值对了。下面是 <code class="highlighter-rouge">runcb()</code> 的代码片段:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">SV</span> <span class="o">*</span>
<span class="n">runcb</span><span class="p">(</span><span class="n">SV</span> <span class="o">*</span><span class="n">sref</span><span class="p">)</span>
<span class="n">CODE</span><span class="o">:</span>
<span class="n">HV</span><span class="o">*</span> <span class="n">plhash</span> <span class="o">=</span> <span class="p">(</span><span class="n">HV</span><span class="o">*</span><span class="p">)(</span><span class="n">SvRV</span><span class="p">(</span><span class="n">sref</span><span class="p">));</span>
<span class="k">const</span> <span class="kt">char</span><span class="o">*</span> <span class="n">key</span> <span class="o">=</span> <span class="s">"call_back"</span><span class="p">;</span>
<span class="n">SV</span><span class="o">*</span> <span class="n">val</span> <span class="o">=</span> <span class="o">*</span><span class="n">hv_fetch</span><span class="p">(</span><span class="n">plhash</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">strlen</span><span class="p">(</span><span class="n">key</span><span class="p">),</span> <span class="mi">0</span><span class="p">);</span>
<span class="n">RETVAL</span> <span class="o">=</span> <span class="n">val</span><span class="p">;</span>
<span class="n">OUTPUT</span><span class="o">:</span> <span class="n">RETVAL</span>
</code></pre>
</div>
<p>这里两个要点,一个是传递进来的哈希引用如何解引用(perl程序里任何时候都不应该直接传递哈希或者数组,而应该传递引用,所以这里直接就研究这步了);一个是 <code class="highlighter-rouge">hv_fetch</code> 的返回值是 <code class="highlighter-rouge">SV**</code> 而不是 <code class="highlighter-rouge">SV*</code>。</p>
<p>发现 XS 语法里比较有意思的一点,就是变量类型转换的时候,大小写的意义。像 <code class="highlighter-rouge">SvRV</code> 就是从 SV 变成 RV,而 <code class="highlighter-rouge">SViv</code> 就是从 IV 变成 SV,基本是谁大写就是转变成谁。</p>
<h1 id="perl-">调用 Perl 函数并获取其返回值</h1>
<p>刚才说到了 descriptor 里的 “call_back” 键的值其实是函数名,所以这一步就试图运行这个 Perl 函数。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">SV</span> <span class="o">*</span>
<span class="n">runcb</span><span class="p">(</span><span class="n">SV</span> <span class="o">*</span><span class="n">sref</span><span class="p">)</span>
<span class="n">CODE</span><span class="o">:</span>
<span class="n">HV</span><span class="o">*</span> <span class="n">plhash</span> <span class="o">=</span> <span class="p">(</span><span class="n">HV</span><span class="o">*</span><span class="p">)(</span><span class="n">SvRV</span><span class="p">(</span><span class="n">sref</span><span class="p">));</span>
<span class="k">const</span> <span class="kt">char</span><span class="o">*</span> <span class="n">key</span> <span class="o">=</span> <span class="s">"call_back"</span><span class="p">;</span>
<span class="n">SV</span><span class="o">*</span> <span class="n">cb</span> <span class="o">=</span> <span class="o">*</span><span class="n">hv_fetch</span><span class="p">(</span><span class="n">plhash</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">strlen</span><span class="p">(</span><span class="n">key</span><span class="p">),</span> <span class="mi">0</span><span class="p">);</span>
<span class="kt">int</span> <span class="n">count</span> <span class="o">=</span> <span class="n">call_sv</span><span class="p">(</span><span class="n">cb</span><span class="p">,</span> <span class="n">G_SCALAR</span><span class="p">);</span>
<span class="n">RETVAL</span> <span class="o">=</span> <span class="n">POPs</span><span class="p">;</span>
<span class="n">OUTPUT</span><span class="o">:</span> <span class="n">RETVAL</span>
</code></pre>
</div>
<p>这里的要点:</p>
<ul>
<li>
<p><code class="highlighter-rouge">call_sv</code> 函数(传递的是函数引用)。在 gmond 源码里用的是 <code class="highlighter-rouge">call_pv</code> 函数(传递的是函数名字符串)。可见原来在代码层这里写起来几乎是一样的,看来定义成写字符串纯粹是作者个人偏好了。</p>
</li>
<li>
<p>这里要给被调用的函数设定上下文,我这里要求返回字符串,就是 <code class="highlighter-rouge">G_SCALAR</code>,还有 <code class="highlighter-rouge">G_VOID</code> 等等,详见 <a href="perldoc.perl.org/perlcall.html">perlcall文档</a>。</p>
</li>
<li>
<p>POPs 操作。<code class="highlighter-rouge">call_sv</code> 函数返回值只代表<strong>被</strong>调用的函数的返回值个数,<strong>被</strong>调用函数的返回值本身,需要另外<em>逐一</em>获取,这个获取就是通过 POPs( 这个是取SV,类似的还有 POPi 等)来完成。</p>
</li>
</ul>
<h1 id="perl--1">给被调用的 Perl 函数传参</h1>
<p>在上面我们可以看到 <code class="highlighter-rouge">call_sv</code> 函数也没有传递参数的地方。那么怎么传递参数给被调用的 Perl 函数呢?</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">SV</span> <span class="o">*</span>
<span class="n">runcb</span><span class="p">(</span><span class="n">SV</span> <span class="o">*</span><span class="n">sref</span><span class="p">,</span> <span class="n">SV</span> <span class="o">*</span><span class="n">argv</span><span class="p">)</span>
<span class="n">CODE</span><span class="o">:</span>
<span class="n">HV</span><span class="o">*</span> <span class="n">plhash</span> <span class="o">=</span> <span class="p">(</span><span class="n">HV</span><span class="o">*</span><span class="p">)(</span><span class="n">SvRV</span><span class="p">(</span><span class="n">sref</span><span class="p">));</span>
<span class="k">const</span> <span class="kt">char</span><span class="o">*</span> <span class="n">key</span> <span class="o">=</span> <span class="s">"callback"</span><span class="p">;</span>
<span class="n">SV</span><span class="o">*</span> <span class="n">cb</span> <span class="o">=</span> <span class="o">*</span><span class="n">hv_fetch</span><span class="p">(</span><span class="n">plhash</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">strlen</span><span class="p">(</span><span class="n">key</span><span class="p">),</span> <span class="mi">0</span><span class="p">);</span>
<span class="n">STRLEN</span> <span class="n">len</span><span class="p">;</span>
<span class="n">PUSHMARK</span><span class="p">(</span><span class="n">SP</span><span class="p">);</span>
<span class="n">XPUSHs</span><span class="p">(</span><span class="n">sv_2mortal</span><span class="p">(</span><span class="n">argv</span><span class="p">));</span>
<span class="n">PUTBACK</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">ret</span> <span class="o">=</span> <span class="n">call_sv</span><span class="p">(</span><span class="n">cb</span><span class="p">,</span> <span class="n">G_SCALAR</span><span class="p">);</span>
<span class="n">SPAGAIN</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">ret</span> <span class="o">!=</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
<span class="n">croak</span><span class="p">(</span><span class="s">"error"</span><span class="p">);</span>
<span class="p">};</span>
<span class="n">SV</span><span class="o">*</span> <span class="n">s</span> <span class="o">=</span> <span class="n">POPs</span><span class="p">;</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"Here: %d %s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">ret</span><span class="p">,</span> <span class="n">SvPV</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="n">len</span><span class="p">));</span>
<span class="n">RETVAL</span> <span class="o">=</span> <span class="n">s</span><span class="p">;</span>
<span class="n">PUTBACK</span><span class="p">;</span>
<span class="n">OUTPUT</span><span class="o">:</span> <span class="n">RETVAL</span>
</code></pre>
</div>
<p>比较复杂啦~~</p>
<p>这里需要有一系列处理 Perl 堆栈的命令来完成传参处理,命令以 <code class="highlighter-rouge">dSP</code> 开头,不过如果编写的是 XS 函数,这步会自动处理可以省略,所以我们这里只需要从 <code class="highlighter-rouge">PUSHMARK</code> 开始。</p>
<p>以 <code class="highlighter-rouge">PUSHMARK</code> 标示开始推入参数到临时区域,然后具体的推入命令是 <code class="highlighter-rouge">XPUSHs</code>(多个就重复推),最后以 <code class="highlighter-rouge">PUTBACK</code> 标示参数推入完成。这时候 Perl 解释器就明白,给下面的 sub 准备的 <code class="highlighter-rouge">@_</code> 已经完毕了,具体大小就是这么多不会再多了。</p>
<p><code class="highlighter-rouge">SPAGAIN</code> 的作用是清理临时区域,因为说不准被调用函数里对临时区域做了什么操作。</p>
<p>同样是 POPs 取出,这里如果直接在 C 代码里 printf 的话,要注意把 SV 转换成 PV,否则是看不对的。</p>
<h1 id="section-2">遍历哈希和返回数组</h1>
<p>前面都是单个变量操作,最后我们来试试哈希遍历,然后返回数组变量。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">AV</span> <span class="o">*</span>
<span class="n">runcb</span><span class="p">(</span><span class="n">SV</span> <span class="o">*</span><span class="n">href</span><span class="p">)</span>
<span class="n">CODE</span><span class="o">:</span>
<span class="n">HV</span><span class="o">*</span> <span class="n">plhash</span> <span class="o">=</span> <span class="p">(</span><span class="n">HV</span><span class="o">*</span><span class="p">)(</span><span class="n">SvRV</span><span class="p">(</span><span class="n">href</span><span class="p">));</span>
<span class="kt">char</span> <span class="o">*</span><span class="n">key</span><span class="p">;</span>
<span class="n">SV</span><span class="o">*</span> <span class="n">sv_value</span><span class="p">;</span>
<span class="n">I32</span> <span class="n">ret</span><span class="p">;</span>
<span class="n">RETVAL</span> <span class="o">=</span> <span class="n">newAV</span><span class="p">();</span>
<span class="n">hv_iterinit</span><span class="p">(</span><span class="n">plhash</span><span class="p">);</span>
<span class="k">while</span> <span class="p">((</span><span class="n">sv_value</span> <span class="o">=</span> <span class="n">hv_iternextsv</span><span class="p">(</span><span class="n">plhash</span><span class="p">,</span> <span class="o">&</span><span class="n">key</span><span class="p">,</span> <span class="o">&</span><span class="n">ret</span><span class="p">)))</span> <span class="p">{</span>
<span class="n">av_push</span><span class="p">(</span><span class="n">RETVAL</span><span class="p">,</span> <span class="n">sv_value</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">OUTPUT</span><span class="o">:</span> <span class="n">RETVAL</span>
</code></pre>
</div>
<p>这里几个要点:</p>
<ul>
<li><code class="highlighter-rouge">runcb()</code> 函数的返回类型要改成 <code class="highlighter-rouge">AV*</code> 了。</li>
<li><code class="highlighter-rouge">RETVAL</code> 需要单独声明赋值才行。</li>
</ul>
<p>写到这里我顺带想到,虽然 Perl5 一直都不对函数传参做什么验证,但是其实 XS 是 C 的自定义语言,所以写 XS 的时候,传参是会自动验证的。Perl5 二十年轮回,今年终于把传参验证给加上了,只能说一代人有一代人的想法啊。。。</p>
给 Kibana 实现百分比统计图表
2014-05-17T00:00:00+08:00
logstash
kibana
elasticsearch
angularjs
http://chenlinux.com/2014/05/17/implement-percentiles-aggregation-on-kibana
<p>kibana 图表类型中有个 stats 类型,返回对应请求的某指定数值字段的数学统计值,包括最大值、最小值、平均值、方差和标准差(当前通过 logstash-1.4.1 分发的 kibana 版本还只支持单列显示,前天,即 5 月 15 日刚<a href="http://www.elasticsearch.org/blog/kibana-3-1/">更新了 Kibana 3.1 版</a>,支持多列同时显示)。这个 stats 图表是利用 Elasticsearch 的 facets 功能来实现的。而在 Elasticsearch 1.0 版本以后,新出现了一个更细致的功能叫 aggregation,按照官方文档所说,会慢慢的彻底替代掉 facets。具体到 1.1 版本的时候, aggregation 里多了一项 percentile,可以具体返回某指定数值字段的区间分布情况。这对日志分析可是大有帮助。对这项功能,Elasticsearch 官方也很得意的专门在博客上写了一篇报道:<a href="http://www.elasticsearch.org/blog/averages-can-dangerous-use-percentile/">Averages can be misleading: try a percentile</a>。</p>
<p>周五晚上下班前,我突然决定试试给 Kibana 加上 percentile 图表类型。因为群里正好携程的同学说到他们仿造 trend 类型做了 stat_trend 图表,我想 percentile 从数据结构到展示方法跟 stats 都很像,应该难度不大,正好作为学习 angularjs 的入手点好了。</p>
<p>花了半天多的时间,基本搞定这件事情,中间几度碰到难题,这里记录一下:</p>
<h1 id="kibana-31--elasticjs-">kibana 3.1 中的 elasticjs 版本</h1>
<p>这是一个非常非常坑爹的地方,kibana/src/vendor/elasticjs/elastic.js 文件开头写着版本号是 <code class="highlighter-rouge">v1.1.1</code>,但是其实它是大半年前(2013-08-14)的。而实际它加上 aggregation 支持的时间是今年的 3 月 16 号,最近版本是 3 月 21 号发布的 ——但是版本号依然是 <code class="highlighter-rouge">v1.1.1</code>!!</p>
<p>我在昨天晚上花了一个多小时慢慢看完了 elasticjs 官网上 v1.1.1 的<a href="http://docs.fullscale.co/elasticjs/ejs.FilterAggregation.html">接口说明</a>,结果其实在 kibana3.1 自带的 elasticjs 上完全不可用。</p>
<h1 id="elasticjs-">elasticjs 新版用法</h1>
<p>随后我替换成了最新的 elasticjs 文件,结果依然不可用,仔细看过文档后发现,新的 elasticjs 只专心处理请求的 DSL,把客户端初始化、配置、收发等事情都交给了 Elasticsearch 官方发布的 elasticsearch.js 来完成。原先版本自带的 elastic-angular-client.js 压根就没用了。</p>
<p>变动大成这样了,居然还不改版本号!?!?</p>
<h1 id="elasticsearchjs-">elasticsearch.js 的多层目录</h1>
<p>下载了 elasticsearch.js 源码后,发现目录里有一个 elasticsearch.angular.client.js 文件,于是我很开心的想,官方考虑的还是很周全的嘛!然后花了一阵功夫在 kibana/src/app/app.js、kibana/src/app/components/require.config.js 等各处添加上了这个 elasticsearch 模块。结果依然不可用。</p>
<p>原来整个 elasticsearch.js 把功能模块化拆分到了很多个不同的多层次的目录里,然后相互之间广泛采用类似 <code class="highlighter-rouge">require('../lib/util/')</code> 这样的语句进行加载。</p>
<p>但是:Kibana 采用的是 requirejs 和 angularjs 合作的模式,整个 js 库的加载过程完全在 kibana/src/app/components/require.config.js 一个文件里定义,你可以看到这个文件里就写了很多 jquery 的子项目文件,但是这些文件都是平铺在 kibana/src/vendor/jquery/ 这个目录里的。</p>
<p>所以,即便在 require.config.js 里写了 elasticsearh 也没用,文件里的 require 语句依然是报错的。而且再往下的压根没法继续添加到 require.config.js 里了,因为太复杂了,肯定得修改 elasticsearch.js 源码的各个文件。</p>
<p>总的来说,就是 elasticsearch.js 不适合跟 requirejs 一起工作。</p>
<hr />
<p>至此,简单更新 js 库然后调用现成接口的计划完全破产。</p>
<p>感谢 Elasticsearch 本身就是一个 RESTful 接口,所以还剩下一个不太漂亮但是确实好用的办法,那就是自己组装请求数据,直接通过 angularjs 内置的 <code class="highlighter-rouge">$http</code> 收发。</p>
<h1 id="aggregationname-">aggregation_name 的限制</h1>
<p>angularjs 的 <code class="highlighter-rouge">$http.post</code> 使用跟 jquery 的 <code class="highlighter-rouge">$.post</code> 非常类似,所以写起来难度不大,确定这个思路之后唯一碰到的问题却是 Elasticsearch 本身的新限制。</p>
<p>目前 Kibana 里都是以 alias 形式来区分每一个子请求的,具体内容是 <code class="highlighter-rouge">var alias = q.alias || q.query;</code>,即在页面上搜索框里写的查询语句或者是搜索框左侧色彩设置菜单里的 <code class="highlighter-rouge">Legend value</code>。</p>
<p>比如我的场景下,<code class="highlighter-rouge">q.query</code> 是 “xff:10.5.16.*“,<code class="highlighter-rouge">q.alias</code> 是”教育网访问”。那么最后发送的请求里这条过滤项的 <code class="highlighter-rouge">facets_name</code> 就叫 “stats_教育网访问”。</p>
<p>同样的写法迁移到 aggregation 上就完全不可解析了。<strong>服务器会返回一条报错说:<code class="highlighter-rouge">aggregation_name</code> 只能是字母、数字、<code class="highlighter-rouge">_</code> 或者 <code class="highlighter-rouge">-</code> 四种。</strong></p>
<p>(这里比较怪的是抓包看到 facets 其实也报错说请求内容解析失败,但是居然同时也返回了结果,只能猜测目前是处在一种兼容状态?)</p>
<p>于是这里稍微修改了一下逻辑,把 <code class="highlighter-rouge">queries</code> 数组的 <code class="highlighter-rouge">_.each</code> 改用 <code class="highlighter-rouge">$.each</code> 来做,这样回调函数里不单返回数组元素,还返回数组下标,下标是一定为数字的,就可以以数组下标作为 <code class="highlighter-rouge">aggregation_name</code> 了。后面处理结果的 <code class="highlighter-rouge">queries.map</code> 同样以下标来获取即可。</p>
<p>目前效果图如下:</p>
<p><img src="/images/uploads/kibana-percentile.png" alt="" /></p>
<p>我的改动已经上传到 <a href="https://github.com/chenryn/kibana/commit/c27b44996bff575886041e0f4f800fda04fbdbde">github</a> 上,欢迎大家一起改进。</p>
<p>目前的问题有两个:图表里的列排序功能不可用,还没找到原因;percents 值还没在 editor.html 里提供自定义办法。</p>
<hr />
<p>2014.05.26 更新: percents 值已经可以自定义</p>
<hr />
<p>2014.06.06 更新: 排序功能可用。原因是 elasticsearch 不管你提交的 percents 带不带小数点,返回值里都会保留小数点后一位,而在 <code class="highlighter-rouge">sortBy</code> 里头,这个小数点就会被理解成 javascript 里获取数据结构键值的意思。所以收到响应后,用 <code class="highlighter-rouge">parseInt</code> 函数干掉小数点就可以了。</p>
用 Graphite 存储 Nagios 数据
2014-05-10T00:00:00+08:00
monitor
nagios
graphite
kibana
http://chenlinux.com/2014/05/10/graphite-grafana-on-nagios
<p>我们都知道 nagios 上可以用 pnp4nagios 来转换 perfdata 成 rrd 图。不过 graphite 以其扩展性及更好的 HTTP 接口目前越来越受欢迎,加上最近刚出来的 grafana 项目(从 LEK 的 Kibana 转化来的),更是让 graphite 的可视化效果也上了一个台阶。</p>
<p>那么怎么用 grafana 来查看我们用 nagios 收集来的监控数据呢?</p>
<p>我在 github 上看到有一个叫 graphios 的项目。不过上面介绍的方法已经比较老了,目前 omd 使用的是 npcmod 的 bulk mode,并不会分别产生 <code class="highlighter-rouge">host-perfdata.$TIMET$</code> 和 <code class="highlighter-rouge">service-perfdata.$TIMET$</code> 文件。所以照着 README 做是没效果的。</p>
<p>最好的办法就是利用 <a href="https://metacpan.org/pod/Net::Graphite">Net::Graphite</a> 模块自己改写 <code class="highlighter-rouge">process-perfdata.pl</code>,把数据直接发给 carbon 进程。不过我懒得动手,目前只是写了几行 perl ,在调用 <code class="highlighter-rouge">process-perfdata.pl</code> 之前,先过一遍 <code class="highlighter-rouge">perfdata.$TIMET</code> 文件,分离出来 host 和 service 两个文件放到新目录里,这样就可以继续走通 graphios 的流程了。(当然性能上比较烂,因为磁盘 IO 翻倍了)</p>
<p>然后是 grafana 部分。grafana 本身基于 kibana 改造而来,所以也是一个纯 js 应用,不过请求 graphite 数据可能涉及跨域 ajax,要求 graphite 的 apache 配置加上几个 Header,这个照着 README 做就可以了。然后不要忘了修改 config.js 里对应的 es 和 graphite 两个服务器地址。</p>
<p>graphite 毕竟数据是以 tree 的唯一格式存在,所以在 grafana 上创建图形时的操作跟 kibana 上不太一样。添加 panel 后,默认是空数据的,然后要在 panel正上方的标题上点击鼠标,选择 <code class="highlighter-rouge">edit</code>,就会出现配置框。</p>
<p>在配置框的 <code class="highlighter-rouge">Metrics</code> 栏选择 <code class="highlighter-rouge">Add query</code>,然后 <code class="highlighter-rouge">select metric</code> 一路选择下去到你想到添加的数值。数值之后点 <code class="highlighter-rouge">+</code> 号还可以添加一些 graphite 计算的值,像平均数啊之类的。这些可以参考 graphite 接口文档。</p>
<p><img src="/images/uploads/add-metric.png" alt="" /></p>
<p>一个简单的效果图如下:</p>
<p><img src="/images/uploads/grafana.png" alt="" /></p>
在 Perl6 脚本中并发执行 ssh 命令
2014-05-04T00:00:00+08:00
perl
perl
perl6
thread
openssh
http://chenlinux.com/2014/05/04/openssh-perl6
<p>前几天翻 Perl6 模块清单,发现没有用作 SSH 的。虽说 Perl6 里可以很方便的用 NativeCall 包装 C/C++ 库,但是 libssh2 本身就不支持我的 kerberos5 认证环境,所以还是只能通过调用系统命令的方式来完成。</p>
<h1 id="thread-">Thread 示例</h1>
<p>说起来 Perl6 近年一直在宣传 Promise 啊,Supply 啊并发编程,但是 API 变化太快,2013 年中期 jnthn 演讲里演示的 <code class="highlighter-rouge">async</code> 用法,现在就直接报这个函数不存在了,似乎改成 <code class="highlighter-rouge">start</code> 了?天知道什么时候又变。所以还是用底层的 Thread 和 Channel 来写。话说其实这还是我第一次写 Thread 呢。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nv">v6</span><span class="p">;</span>
<span class="nv">class</span> <span class="nv">OpenSSH</span> <span class="p">{</span>
<span class="nv">has</span> <span class="nv">$</span><span class="err">!</span><span class="nv">user</span> <span class="o">=</span> <span class="s">'root'</span><span class="p">;</span>
<span class="nv">has</span> <span class="nv">$</span><span class="err">!</span><span class="nv">port</span> <span class="o">=</span> <span class="mi">22</span><span class="p">;</span>
<span class="nv">has</span> <span class="nv">$</span><span class="err">!</span><span class="nv">ssh</span> <span class="o">=</span> <span class="s">"ssh -oStrictHostKeyChecking=no -l{$!user} -p{$!port} "</span><span class="p">;</span>
<span class="nv">multi</span> <span class="nv">method</span> <span class="nb">exec</span><span class="p">(</span><span class="nv">$host</span><span class="p">,</span> <span class="nv">$cmd</span><span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$out</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$shell</span> <span class="o">=</span> <span class="nv">$</span><span class="err">!</span><span class="nv">ssh</span> <span class="o">~</span> <span class="nv">$host</span> <span class="o">~</span> <span class="s">' '</span> <span class="o">~</span> <span class="nv">$cmd</span><span class="p">;</span>
<span class="nv">try</span> <span class="p">{</span> <span class="nv">$out</span> <span class="o">=</span> <span class="nv">qqx</span><span class="p">{</span> <span class="nv">$shell</span> <span class="p">}</span><span class="o">.</span><span class="nb">chomp</span> <span class="p">}</span>
<span class="nv">CATCH</span> <span class="p">{</span> <span class="nv">note</span><span class="p">(</span><span class="s">"Failed: $!"</span><span class="p">)</span> <span class="p">};</span>
<span class="k">return</span> <span class="nv">$out</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">multi</span> <span class="nv">method</span> <span class="nb">exec</span><span class="p">(</span><span class="nv">@hosts</span><span class="p">,</span> <span class="nv">$cmd</span><span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$c</span> <span class="o">=</span> <span class="nv">Channel</span><span class="o">.</span><span class="k">new</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@t</span> <span class="o">=</span> <span class="nv">@hosts</span><span class="o">.</span><span class="nb">map</span><span class="p">({</span>
<span class="nv">Thread</span><span class="o">.</span><span class="nv">start</span><span class="p">({</span>
<span class="k">my</span> <span class="nv">$r</span> <span class="o">=</span> <span class="nv">$</span><span class="err">.</span><span class="nv">exec</span><span class="p">(</span><span class="nv">$_</span><span class="p">,</span> <span class="nv">$cmd</span><span class="p">);</span>
<span class="nv">$c</span><span class="o">.</span><span class="nb">send</span><span class="p">(</span><span class="nv">$r</span><span class="p">);</span>
<span class="p">})</span>
<span class="p">});</span>
<span class="nv">@t</span><span class="o">>>.</span><span class="nv">finish</span><span class="p">;</span>
<span class="k">return</span> <span class="nv">@hosts</span><span class="o">.</span><span class="nb">map</span><span class="p">:</span> <span class="p">{</span> <span class="nv">$c</span><span class="o">.</span><span class="nv">receive</span> <span class="p">};</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">my</span> <span class="nv">$ssh</span> <span class="o">=</span> <span class="nv">OpenSSH</span><span class="o">.</span><span class="k">new</span><span class="p">(</span><span class="nv">user</span> <span class="o">=></span> <span class="s">'root'</span><span class="p">);</span>
<span class="nv">say</span> <span class="nv">$ssh</span><span class="o">.</span><span class="nb">exec</span><span class="p">(</span><span class="s">'10.4.1.21'</span><span class="p">,</span> <span class="s">'uptime'</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">@hosts</span> <span class="o">=</span> <span class="s">'10.4.1.21'</span> <span class="nv">xx</span> <span class="mi">5</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@ret</span> <span class="o">=</span> <span class="nv">$ssh</span><span class="o">.</span><span class="nb">exec</span><span class="p">(</span><span class="nv">@hosts</span><span class="p">,</span> <span class="s">'sleep 3;echo $$'</span><span class="p">);</span>
<span class="nv">say</span> <span class="nv">@ret</span><span class="o">.</span><span class="nv">perl</span><span class="p">;</span>
</code></pre>
</div>
<p>很简陋的代码。首先一个是要确认 ssh 不用密码登陆,因为没有写 Expect;其次是没用 ThreadPool,所以并发操作不能太猛,会扭着腰的。</p>
<p>这里演示了几个地方:</p>
<ul>
<li>class 的定义和 attr 的定义和<a href="http://doc.perl6.org/language/classtut">用法</a></li>
<li>
<p>try-catch 的用法</p>
<p>也可以不写 try,直接 <code class="highlighter-rouge">CATCH {}</code></p>
</li>
<li>
<p>qqx{} 的用法</p>
<p>这是变动比较大的地方,<code class="highlighter-rouge">qqx</code> 后面只能用 <code class="highlighter-rouge"><span class="p">{}</span></code> 不能用其他字符对了。Perl6 提供另外的 <code class="highlighter-rouge">shell()</code> 指令,返回 <code class="highlighter-rouge">Proc::Status</code> 对象。<br />
不过这个对象其实也就是个状态码,不包括标准输出、错误输出什么的。</p>
</li>
<li>字符串连接符 ~ 的用法</li>
<li>multi method 的定义和用法</li>
<li><a href="http://doc.perl6.org/type/Method#signature">函数 signature</a> 的定义和用法,可选参数和命名参数的定义和用法见下一小节。</li>
<li>
<p><code class="highlighter-rouge">>></code> 操作符的用法</p>
<p>这里其实相当于是 <code class="highlighter-rouge">.finish for @t</code>。这个怪怪的操作符据说可以在可能的时候自动线程化数组操作,所以返回顺序不会跟<code class="highlighter-rouge">.map</code>一样。</p>
</li>
<li>
<p>xx 操作符的用法</p>
<p>Perl5 里有 <code class="highlighter-rouge">x</code> 操作符,Perl6 里又增加了 <code class="highlighter-rouge">xx</code>、 <code class="highlighter-rouge">X</code> 和 <code class="highlighter-rouge">Z</code> 等操作符。<br />
分别是<a href="http://doc.perl6.org/language/operators#infix_xx">字符扩展成数组</a>、<a href="http://doc.perl6.org/language/operators#infix_X">数组扩展成多维数组</a>和<a href="http://doc.perl6.org/language/operators#infix_Z">多数组压缩单个数组</a>(也就是zip操作)。</p>
</li>
<li>
<p>Channel 和 Thread 对象的用法</p>
<p>在 roast 测试集里,只有 thread 和 lock 的<a href="https://github.com/perl6/roast/blob/master/S17-lowlevel/lock.t">测试用例</a>。<br />
semaphore 其实也支持(因为 MoarVM 是基于 libuv 的嘛,libuv 支持它当然也支持),但是连测试用例都没写……</p>
</li>
</ul>
<p>默认的并发编程会采用 <code class="highlighter-rouge">ThreadPoolScheduler</code> 类,稍微看了一下,默认设置的线程数是 16。考虑下一步是仿照该类完善我的小脚本呢,还是重新学习一下 <code class="highlighter-rouge">Supply</code> 或者 <code class="highlighter-rouge">Promise</code> 看看到底怎么用。</p>
<p>有兴趣用 libssh2 的童鞋,可以学习一下 <a href="https://github.com/jnthn/zavolaj">NativeCall</a> 的用法。</p>
<h1 id="threadpoolscheduler-">ThreadPoolScheduler 示例</h1>
<p>根据 <a href="https://github.com/perl6/specs/blob/master/S17-concurrency.pod">S17-concurrency 文档</a> 的内容,改写了几行脚本,实现了 ThreadPool 的效果:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">multi</span> <span class="nv">method</span> <span class="nb">exec</span><span class="p">(</span><span class="nv">@hosts</span><span class="p">,</span> <span class="nv">$cmd</span><span class="p">,</span> <span class="p">:</span><span class="nv">$parallel</span> <span class="o">=</span> <span class="mi">16</span><span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$c</span> <span class="o">=</span> <span class="nv">Channel</span><span class="o">.</span><span class="k">new</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$s</span> <span class="o">=</span> <span class="nv">ThreadPoolScheduler</span><span class="o">.</span><span class="k">new</span><span class="p">(</span><span class="nv">max_threads</span> <span class="o">=></span> <span class="nv">$parallel</span><span class="p">);</span>
<span class="nv">@hosts</span><span class="o">.</span><span class="nb">map</span><span class="p">({</span>
<span class="nv">$s</span><span class="o">.</span><span class="nv">cue</span><span class="p">({</span>
<span class="k">my</span> <span class="nv">$r</span> <span class="o">=</span> <span class="nv">$</span><span class="err">.</span><span class="nv">exec</span><span class="p">(</span><span class="nv">$_</span><span class="p">,</span> <span class="nv">$cmd</span><span class="p">);</span>
<span class="nv">$c</span><span class="o">.</span><span class="nb">send</span><span class="p">(</span><span class="nv">$r</span><span class="p">);</span>
<span class="p">})</span>
<span class="p">});</span>
<span class="k">return</span> <span class="nv">@hosts</span><span class="o">.</span><span class="nb">map</span><span class="p">:</span> <span class="p">{</span> <span class="nv">$c</span><span class="o">.</span><span class="nv">receive</span> <span class="p">};</span>
<span class="p">}</span>
</code></pre>
</div>
<p>这里把默认并发值改成了 16,跟 Rakudo 保持一致。如果不需要可调的话,这里其实可以直接写成 <code class="highlighter-rouge">$*SCHEDULER.cue({})</code>。</p>
<p>然后调用方法也对应修改一下,考虑到辨识度,把并发值改成了命名参数。调用方法如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">my</span> <span class="nv">@hosts</span> <span class="o">=</span> <span class="nv">slurp</span><span class="p">(</span><span class="s">'iplist.txt'</span><span class="p">)</span><span class="o">.</span><span class="nv">lines</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@ret</span> <span class="o">=</span> <span class="nv">$ssh</span><span class="o">.</span><span class="nb">exec</span><span class="p">(</span><span class="nv">@hosts</span><span class="p">,</span> <span class="s">'sleep 3;echo $$'</span><span class="p">,</span> <span class="p">:</span><span class="nv">parallel</span><span class="p">(</span><span class="mi">5</span><span class="p">));</span>
</code></pre>
</div>
<p>运行可以看到,虽然 iplist.txt 里放了 40 个ip,但是并发的 ssh 只有 5 个。</p>
<h1 id="promise-">Promise 示例</h1>
<p>继续,S17 内容下一节是 Promise,之前博客里已经提过几次 Perl5 的 <a href="https://metacpan.org/pod/Promises">Promises 模块</a> 或者类似的东西(比如 <a href="/2014/01/22/explain-mojo-ioloop-delay-testing">Mojo::IOLoop::Delay</a> ),包括 JavaScript 等也有一样的名字。</p>
<p>不过 Perl5 的 Promises 思路参照的是 Scala,语法则偏向 nodejs 和 golang(都用一个叫 <code class="highlighter-rouge">defer</code> 的指令来创建 Promises 对象),写起来跟 Perl6 的原生 Promise 差距较大。</p>
<p>考虑 ssh 这个场景可能不太用的上 Promise 的 <code class="highlighter-rouge">.in</code>、<code class="highlighter-rouge">.then</code>、<code class="highlighter-rouge">.anyof</code> 之类的流程控制(尤其 <code class="highlighter-rouge">.in</code> 这个还不一定能用,因为 Promise 底层也是用的 <code class="highlighter-rouge">$*SCHEDULER.cue()</code>,而这个在 MoarVM 上目前还不支持 :in/:at/:every 等参数),就直接展示最简单的并发了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">multi</span> <span class="nv">method</span> <span class="nb">exec</span><span class="p">(</span><span class="nv">@hosts</span><span class="p">,</span> <span class="nv">$cmd</span><span class="p">,</span> <span class="p">:</span><span class="nv">$parallel</span> <span class="o">=</span> <span class="mi">16</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$</span><span class="err">*</span><span class="nv">SCHEDULER</span> <span class="o">=</span> <span class="nv">ThreadPoolScheduler</span><span class="o">.</span><span class="k">new</span><span class="p">(</span><span class="nv">max_threads</span> <span class="o">=></span> <span class="nv">$parallel</span><span class="p">);</span>
<span class="nv">await</span> <span class="nv">@hosts</span><span class="o">.</span><span class="nb">map</span><span class="p">:</span> <span class="p">{</span>
<span class="nv">start</span> <span class="p">{</span>
<span class="nv">$</span><span class="err">.</span><span class="nv">exec</span><span class="p">(</span><span class="nv">$_</span><span class="p">,</span> <span class="nv">$cmd</span><span class="p">);</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="p">}</span>
</code></pre>
</div>
<p>简单来说,就是每个 <code class="highlighter-rouge">start {&c}</code> 创建一个 Promise 对象,根据 &c 的返回值自动作 <code class="highlighter-rouge">$p.keep($result)</code> 或 <code class="highlighter-rouge">$p.break(Exception)</code>。然后 <code class="highlighter-rouge">await(*@p)</code> 回收全部 Promise 的结果。</p>
<p>这里直接修改了 <code class="highlighter-rouge">$*SCHEDULER</code> ,这是一个全局变量,即当前进程的调度方式。Promise 类默认就采用这个变量。如果想跟上一小节一样使用 <code class="highlighter-rouge">$s</code>,那这里就不能用 <code class="highlighter-rouge">start {}</code> 而是要用 <code class="highlighter-rouge">Promise.start({}, $s)</code>。显然写起来不怎么漂亮。</p>
<h1 id="supply-">Supply 示例</h1>
<p>Supply 是响应式编程,类似 Java 里的 Reactive 概念。应该适合的是一件事情多个进程重复做。场景不太对,二来目前 S17 也不全,就不写了。</p>
Perl6 的 YAML::Dumper 模块
2014-04-24T00:00:00+08:00
perl
rakudo
moarvm
perl6
yaml
sqlite
http://chenlinux.com/2014/04/24/yaml-dump-pm6
<p>这两天决定试一把 Perl6,因为<a href="http://www.php-oa.com">扶凯</a>兄已经把还没有正式发行 Rakudo Star 包的 MoarVM 编译打包好了,所以可以跳过这步直接进入模块安装。当然,源码编译本身也没有太大难度,只不过从 github 下源码本身耗时间比较久而已。</p>
<p>既然木有 Star 包,那么安装好 MoarVM 上的 Rakudo 后我们就有必要先自己把 panda 之类的工具编译出来。这一步需要注意一下你的 <code class="highlighter-rouge">@*INC</code> 路径和实际的 <code class="highlighter-rouge">$PERL6LIB</code> 路径,已经编译之后的 panda 存在的 <code class="highlighter-rouge">$PATH</code> 是不是都正确,如果不对的修改一下 <code class="highlighter-rouge">~/.bashrc</code> 就好了。</p>
<p>我的尝试迁移对象是一个很简单的 Puppet 的 ENC 脚本,只涉及 SQLite 的读取,以及 YAML 格式的输出。通过 <code class="highlighter-rouge">panda install DBIish</code> 命令即可安装好 DBIish 模块。</p>
<p>脚本本身修改起来难度不大,结果如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/env perl6</span>
<span class="k">use</span> <span class="nv">v6</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">DBIish</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">YAML</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$base_dir</span> <span class="o">=</span> <span class="s">"/etc/puppet/webui"</span><span class="p">;</span>
<span class="c1"># 函数在 Perl6 中依然使用 sub 关键字定义,不过有个超酷的特性是 multi sub</span>
<span class="c1"># 脚本中没有用到,但是在 YAML::Dumper 中遍地都是,这里也提一句。</span>
<span class="c1"># MAIN 函数在 Perl6 里可以直接用 :$opt 命令参数起 getopt 的作用</span>
<span class="c1"># 不过 ENC 脚本就是直接传一个主机名,用不上这个超酷的特性</span>
<span class="k">sub </span><span class="nf">MAIN</span><span class="p">($node) {</span>
<span class="c1"># connect 方法接收参数选项是 |%opts,所以可以把哈希直接平铺写</span>
<span class="c1"># 这个 | 的用法一个月前在《Using Perl6》里看到过</span>
<span class="k">my</span> <span class="nv">$dbh</span> <span class="o">=</span> <span class="nv">DBIish</span><span class="o">.</span><span class="nb">connect</span><span class="p">(</span> <span class="s">'SQLite'</span><span class="p">,</span> <span class="nv">database</span> <span class="o">=></span> <span class="s">"{$base_dir}/node_info.db"</span> <span class="p">);</span>
<span class="k">my</span> <span class="nv">$sth</span> <span class="o">=</span> <span class="nv">$dbh</span><span class="o">.</span><span class="nv">prepare</span><span class="p">(</span><span class="s">"select * from node_info where node_fqdn = ?"</span><span class="p">);</span>
<span class="nv">$sth</span><span class="o">.</span><span class="nv">execute</span><span class="p">(</span><span class="s">"$node"</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$ret</span> <span class="o">=</span> <span class="nv">$sth</span><span class="o">.</span><span class="nv">fetchrow_hashref</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$res</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span> <span class="o">!</span><span class="nv">$ret</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$res</span> <span class="o">=</span> <span class="p">{</span>
<span class="c1"># Perl5 的 qw() 在 Perl6 里直接写成 <> 。也不用再通过 [] 来指明是引用</span>
<span class="nv">classes</span> <span class="o">=></span> <span class="o"><</span><span class="nv">puppetd</span> <span class="nv">repos</span><span class="o">></span><span class="p">,</span>
<span class="nv">environment</span> <span class="o">=></span> <span class="s">'testing'</span><span class="p">,</span>
<span class="p">};</span>
<span class="p">}</span>
<span class="k">else</span> <span class="p">{</span>
<span class="nv">$res</span> <span class="o">=</span> <span class="p">{</span>
<span class="nv">environment</span> <span class="o">=></span> <span class="nv">$ret</span><span class="p">{</span><span class="s">'environment'</span><span class="p">},</span>
<span class="nv">parameters</span> <span class="o">=></span> <span class="p">{</span> <span class="nv">role</span> <span class="o">=></span> <span class="nv">$ret</span><span class="p">{</span><span class="s">'role'</span><span class="p">}</span> <span class="p">},</span>
<span class="nv">classes</span> <span class="o">=></span> <span class="p">{},</span>
<span class="p">};</span>
<span class="c1"># 这个 for 的用法,在 Perl5 的 Text::Xslate 模板里就用过</span>
<span class="k">for</span> <span class="nb">split</span><span class="p">(</span><span class="s">','</span><span class="p">,</span> <span class="nv">$ret</span><span class="p">{</span><span class="s">'classes'</span><span class="p">})</span> <span class="o">-></span> <span class="nv">$class</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$class</span> <span class="ow">eq</span> <span class="s">'nginx'</span> <span class="p">)</span> <span class="p">{</span>
<span class="c1"># 这个 <== 符号指明数据流方向,完全可以把数组倒过来,然后用 ==> 写这行</span>
<span class="c1"># 如果不习惯这种流向操作符的,可以用,号,反正不能跟 Perl5 那样啥都不写</span>
<span class="c1"># 这里比较怪的一点是我试图把这么长的一句分成多行写,包括每行后面加\,我看到 YAML 代码里就用\分行了,但是我这就会报错</span>
<span class="c1"># Perl6 的正则变化较大,这里 /^#/ 要写成 /^'#'/ 或者 /^\x23/</span>
<span class="c1"># 正则 // 前面不加 m// 不会立刻开始匹配</span>
<span class="c1"># 原先的 s///g 可以写作 s:g///,也可以写作对象式的 .subst(m//, '', :g),. 前面为空就是默认的 $_</span>
<span class="c1"># 捕获的数据存在 @() 数组里,也可以用 $/[i] 的形式获取</span>
<span class="c1"># 字符串内插时,不再写作 ${*},而是 {$*} 的形式</span>
<span class="c1"># 命名捕获这里没用上,写个示例:</span>
<span class="c1"># $str ~~ /^(\w+?)$<laststr>=(\w ** 4)\w$/;</span>
<span class="c1"># $/<laststr>.chomp.say;</span>
<span class="c1"># 注意里面的 \w{4} 变成了 \w ** 4</span>
<span class="k">my</span> <span class="nv">@needs</span> <span class="o"><==</span> <span class="nb">map</span> <span class="p">{</span> <span class="o">.</span><span class="nv">subst</span><span class="p">(</span><span class="sr">m/^(.+)\:(\d+)$/</span><span class="p">,</span> <span class="s">"{$/[0]} max_fails=30 weight={$/[1]}"</span><span class="p">,</span> <span class="p">:</span><span class="nv">g</span><span class="p">)</span> <span class="p">}</span> <span class="o"><==</span> <span class="nb">grep</span> <span class="p">{</span> <span class="o">!</span><span class="sr">m/^\x23/</span> <span class="p">}</span> <span class="o"><==</span> <span class="nb">split</span><span class="p">(</span><span class="s">','</span><span class="p">,</span> <span class="nv">$ret</span><span class="p">{</span><span class="s">'extstr'</span><span class="p">});</span>
<span class="nv">$res</span><span class="p">{</span><span class="s">'classes'</span><span class="p">}{</span><span class="s">'nginx'</span><span class="p">}{</span><span class="s">'iplist'</span><span class="p">}</span> <span class="o">=</span> <span class="nv">@needs</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">else</span> <span class="p">{</span>
<span class="c1"># Perl5 的 undef 不再使用,可以使用 Nil 或者 Any 对象</span>
<span class="nv">$res</span><span class="p">{</span><span class="s">'classes'</span><span class="p">}{</span><span class="nv">$class</span><span class="p">}</span> <span class="o">=</span> <span class="nv">Nil</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="nv">$dbh</span><span class="o">.</span><span class="nv">disconnect</span><span class="p">();</span>
<span class="c1"># 这个 dump 就是 YAML 模块导出的函数</span>
<span class="c1"># Perl6 的模块要导出函数不再需要 Exporter 那样,直接用 our sub dump($obj) {} 就可以了</span>
<span class="nv">say</span> <span class="nb">dump</span><span class="p">(</span><span class="nv">$res</span><span class="p">);</span>
<span class="p">};</span>
</code></pre>
</div>
<p>但是麻烦的是 YAML 模块本身,这个模块是 ingydotnet 在好几年前草就,后来就没管了,实际现在压根跑不起来。花了半天时间,一边学习一边修改,总算修改正常了。主要涉及了 <code class="highlighter-rouge">Attribute</code> 对象,<code class="highlighter-rouge">Nil</code> 对象,<code class="highlighter-rouge">twigls</code> 前缀符,<code class="highlighter-rouge">:exists</code> 定义几个概念,以及 YAML 格式本身的处理逻辑。</p>
<p>YAML 模块修改对比如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="gh">diff --git a/lib/YAML/Dumper.pm b/lib/YAML/Dumper.pm
index d7a7981..ec47341 100644
</span><span class="gd">--- a/lib/YAML/Dumper.pm
</span><span class="gi">+++ b/lib/YAML/Dumper.pm
</span><span class="gu">@@ -2,16 +2,16 @@ use v6;
</span> class YAML::Dumper;
has $.out = [];
<span class="gd">-has $.seen is rw = {};
</span><span class="gi">+has $.seen = {};
</span> has $.tags = {};
has $.anchors = {};
has $.level is rw = 0;
<span class="gd">-has $.id is rw = 1;
</span><span class="gi">+has $.id = 1;
</span> has $.info = [];
method dump($object) {
$.prewalk($object);
<span class="gd">- $.seen = {};
</span><span class="gi">+ $!seen = {};
</span> $.dump_document($object);
return $.out.join('');
}
<span class="gu">@@ -45,11 +45,11 @@ method dump_collection($node, $kind, $function) {
</span>
method check_special($node) {
my $first = 1;
<span class="gd">- if $.anchors.exists($node.WHICH) {
- if $.anchors.exists($node.WHICH) {
</span><span class="gi">+ if $.anchors{$node.WHICH}:exists {
</span> push $.out, ' ', '&' ~ $.anchors{$node.WHICH};
$first = 0;
}
<span class="gd">- if $.tags.exists($node.WHICH) {
</span><span class="gi">+ if $.tags{$node.WHICH}:exists {
</span> push $.out, ' ', '!' ~ $.tags{$node.WHICH};
$first = 0;
}
<span class="gu">@@ -64,7 +64,7 @@ method indent($first) {
</span> return;
}
if $.info[*-1]<kind> eq 'seq' && $.info[*-2]<kind> eq 'map' {
<span class="gd">- $seq_in_map = 1;
</span><span class="gi">+ $seq_in_map = 0;
</span> }
}
push $.out, "\n";
<span class="gu">@@ -155,7 +155,8 @@ method dump_object($node, $type) {
</span> $.tags{$repr.WHICH} = $type;
for $node.^attributes -> $a {
my $name = $a.name.substr(2);
<span class="gd">- my $value = pir::getattribute__PPs($node, $a.name); #RAKUDO
</span><span class="gi">+ #my $value = pir::getattribute__PPs($node, $a.name); #RAKUDO
+ my $value = $a.get_value($node); #for non-parrot
</span> $repr{$name} = $value;
}
$.dump_node($repr);
</code></pre>
</div>
<p>这里的 <code class="highlighter-rouge">$.seen</code> 和 <code class="highlighter-rouge">$!seen</code> 是不是晕掉了?其实 <code class="highlighter-rouge">$.seen</code> 就相当于先声明了 <code class="highlighter-rouge">$!seen</code> 后再自动创建一个 <code class="highlighter-rouge">method seen() { return $!seen }</code>。</p>
<p>另一处是 <code class="highlighter-rouge">pir::getattribute__PPs()</code> 函数,pir 是 parrot 上的语言,而 MoarVM 和 JVM 上都是先实现了一个 nqp 再用 nqp 写 Perl6,不巧的是这个 pir 里的 <code class="highlighter-rouge">getattribute__PPs()</code> 刚好至今还没有对应的 nqp 方法。(在 pir2nqp.todo 文件里可见)</p>
<p>所以只能用高级的 Perl6 语言来做了。</p>
<p>总的来说,这个 yaml-pm6 代码里很多地方都是试来试去,同样的效果不同的写法,又比如 <code class="highlighter-rouge">.WHICH</code> 和 <code class="highlighter-rouge">.WHAT.perl</code> 也是混用。<br />
而且我随手测试了一下,即使在 parrot 上,用 <code class="highlighter-rouge">pir::getattribute__PPs</code> 的速度也比 <code class="highlighter-rouge">Attribute.get_value</code> 还差点点。</p>
<hr />
<p>最后提一句,目前 ENC 脚本在 perl5、perl6-m、perl6-p、perl6-j 上的运行时间大概分别是 0.13、1.5、2.8、12s。MoarVM 还差 Perl5 十倍,领先 parrot 一倍。不过 JVM 本身启动时间很长,这里不好因为一个短时间脚本说它太慢。</p>
<p>另外还试了一下如果把我修改过的 YAML::Dumper 类直接写在脚本里运行,也就是不编译成 moarvm 模块,时间大概是 2.5s,比 parrot 模块还快点点。</p>
<p>不过如何把 perl6 脚本本身编译成 moarvm 的 bytecode 格式运行还没有研究出来,直接 <code class="highlighter-rouge">perl6-m --target=mbc --output=name.moarvm name.pl6</code> 得到的文件运行 <code class="highlighter-rouge">moar name.moarvm</code> 的结果运行会内存报错。</p>
TCP Fast Open 测试(2)
2014-04-21T00:00:00+08:00
linux
tcpdump
httping
systemtap
http://chenlinux.com/2014/04/21/tcp-fastopen-2
<p>接上篇。</p>
<p>18 日提到采用 wireshark 而不是 tcpdump 来抓取数据。wireshark 会自动把一些数据解释成可读的内容,于是看到其实在每次 httping 发出请求的时候,第一个 SYN 包后面都有附加了 TCP FASTOPEN COOKIE 请求:</p>
<p><img src="/images/uploads/foc-req.png" alt="" /></p>
<p>于是回头重新好好读了一下 TFO 的原理,发现自己对 TFO 的理解是有问题的 - 原先我以为在 SYN 里是可以直接带上请求数据的 - 而这很容易被攻击。实际上的流程应该是:</p>
<ol>
<li>客户端发送 SYN 包,包尾加的是一个 FOC 请求,只有 4 字节。</li>
<li>服务器端收到 FOC 请求,验证后根据来源 IP 地址生成 COOKIE(8 字节),将这个 COOKIE 加载 SYN+ACK 包的末尾发送回去。</li>
<li>客户端缓存住获取到的 COOKIE 可以给下一次使用。</li>
<li>下一次请求开始,客户端发送 SYN 包,这时候包后面带上缓存的 COOKIE,然后就是要正式发送的数据。</li>
<li>服务器端验证 COOKIE 正确,将数据交给上层应用处理得到响应结果,然后在发送 SYN+ACK 时,不再等待客户端的 ACK 确认,即开始发送响应数据。</li>
</ol>
<p>示图如下:</p>
<p><img src="/images/uploads/tfo.jpg" alt="" /></p>
<p>所以可以总结两点:</p>
<ol>
<li>第一次请求是不会有时间节约的效果的,测试至少要 <code class="highlighter-rouge">httping -F -c 2</code>。</li>
<li>从第二次开始节约的时间可以认为是第一个来回,httping 本身是个 HEAD 请求,可以认为是 50% 的节约。</li>
</ol>
<p>但是用 <code class="highlighter-rouge">-c 2</code> 运行依然没有看到 RTT 变化。这时候用<code class="highlighter-rouge">stap 'probe kernel.function("tcp_fastopen_cookie_gen") {printf("%d\n", $foc->len)}'</code> 命令发现这个最重要的生成 COOKIE 的函数(net/ipv4/tcp_fastopen.c里)居然一直没有被触发!</p>
<p>认真阅读了一下调用这个函数的 <code class="highlighter-rouge">tcp_fastopen_check</code> 函数(net/ipv4/tcp_ipv4.c里),原来前面首先有一步检查 sysctl 的逻辑:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">if</span> <span class="p">((</span><span class="n">sysctl_tcp_fastopen</span> <span class="o">&</span> <span class="n">TFO_SERVER_ENABLE</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span> <span class="o">||</span>
<span class="n">fastopenq</span> <span class="o">==</span> <span class="nb">NULL</span> <span class="o">||</span> <span class="n">fastopenq</span><span class="o">-></span><span class="n">max_qlen</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">false</span><span class="p">;</span>
</code></pre>
</div>
<p>这个 <code class="highlighter-rouge">TFO_SERVER_ENABLE</code> 常量是 2。而我电脑默认的 <code class="highlighter-rouge">net.ipv4.tcp_fastopen</code> 值是 1。1 只开启客户端支持 TFO,所以这里要改成 2(或者 3,如果你不打算把客户端搬到别的主机上测试的话)。</p>
<p>重新开始 httping 测试,RTT 依然没有缩短。这时候的 stap 命令发现 <code class="highlighter-rouge">tcp_fastopen_cookie_gen</code> 函数虽然触发了,但是函数里真正干活的这段逻辑依然没有触发(即 <code class="highlighter-rouge">crypto_cipher_encrypt_one</code>):</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="kt">void</span> <span class="nf">tcp_fastopen_cookie_gen</span><span class="p">(</span><span class="n">__be32</span> <span class="n">addr</span><span class="p">,</span> <span class="k">struct</span> <span class="n">tcp_fastopen_cookie</span> <span class="o">*</span><span class="n">foc</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">__be32</span> <span class="n">peer_addr</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span> <span class="n">addr</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span> <span class="p">};</span>
<span class="k">struct</span> <span class="n">tcp_fastopen_context</span> <span class="o">*</span><span class="n">ctx</span><span class="p">;</span>
<span class="n">rcu_read_lock</span><span class="p">();</span>
<span class="n">ctx</span> <span class="o">=</span> <span class="n">rcu_dereference</span><span class="p">(</span><span class="n">tcp_fastopen_ctx</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">ctx</span><span class="p">)</span> <span class="p">{</span>
<span class="n">crypto_cipher_encrypt_one</span><span class="p">(</span><span class="n">ctx</span><span class="o">-></span><span class="n">tfm</span><span class="p">,</span>
<span class="n">foc</span><span class="o">-></span><span class="n">val</span><span class="p">,</span>
<span class="p">(</span><span class="n">__u8</span> <span class="o">*</span><span class="p">)</span><span class="n">peer_addr</span><span class="p">);</span>
<span class="n">foc</span><span class="o">-></span><span class="n">len</span> <span class="o">=</span> <span class="n">TCP_FASTOPEN_COOKIE_SIZE</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">rcu_read_unlock</span><span class="p">();</span>
<span class="p">}</span>
</code></pre>
</div>
<p>我试图通过 <code class="highlighter-rouge">stap 'probe kernel.function("tcp_fastopen_cookie_gen"){printf("%s\n", $$locals$$)}'</code> 来查看这个 <code class="highlighter-rouge">ctx</code> 是什么内容。输出显示 ctx 结构里的元素值都是问号。</p>
<p>目前就卡在这里。</p>
<p>为了验证除了这步没有其他问题,我”野蛮”的通过 systemtap 修改了一下 <code class="highlighter-rouge">tcp_fastopen_cookie_gen</code> 里的变量。命令如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>stap 'probe kernel.function("tcp_fastopen_cookie_gen") { $foc->len = 8 }'
</code></pre>
</div>
<p>赋值为 8,就是 <code class="highlighter-rouge">TCP_FASTOPEN_COOKIE_SIZE</code> 常量的值。</p>
<p>然后再运行测试,就发现 httping 的第二次运行的 RTT 时间减半了(最后那个 F 应该就是标记为 Fastopen 的意思吧)!可见目前问题就出在这里。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ httping -F -g http://192.168.0.100 -c 2
PING 192.168.0.100:80 (/url):
connected to 192.168.0.100:80 (154 bytes), seq=0 time= 45.60 ms
connected to 192.168.0.100:80 (154 bytes), seq=1 time= 23.43 ms F
--- http://192.168.0.100/url ping statistics ---
2 connects, 2 ok, 0.00% failed, time 2069ms
round-trip min/avg/max = 23.4/34.5/45.6 ms
</code></pre>
</div>
<p><strong>注:上面这个强制赋值 <code class="highlighter-rouge">foc->len</code> 没有改变其实 <code class="highlighter-rouge">foc->val</code> 是空的事实,所以只能是测试验证一下想法,真用的话多客户端之间会乱套的。</strong></p>
TCP Fast Open 测试(1)
2014-04-16T00:00:00+08:00
linux
nginx
systemtap
http://chenlinux.com/2014/04/16/tcp-fastopen-1
<p><strong>首先,这是一个未完成的测试。</strong></p>
<p>新闻上大家都知道,Nginx从1.5.8开始支持fastopen参数,Linux从3.5开始支持fastopen特性,并在3.10开始默认开启。</p>
<p>httping是一个模拟ping输出的http请求客户端。从1.5开始支持发送fastopen请求,目前版本是2.3.4。</p>
<p>我在 fedora 20 (内核3.13版) 上编译了 nginx 1.5.13,yum 安装了 httping 2.3.3版。</p>
<p>开两个终端,一个运行tcpdump,然后另一个运行httping如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>httping -F -g http://www.google.com.hk/url -c 1
</code></pre>
</div>
<p>这时候看到前一个终端的输出是这样的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[chenlin.rao@com21-100 tfo]$ sudo tcpdump -i p5p1 -vvnxXs 0 tcp port 80
tcpdump: listening on p5p1, link-type EN10MB (Ethernet), capture size 65535 bytes
20:40:15.034486 IP (tos 0x0, ttl 64, id 52862, offset 0, flags [DF], proto TCP (6), length 147)
10.2.5.100.40699 > 74.125.128.199.http: Flags [S], cksum 0xbb34 (correct), seq 3616187260:3616187335, win 29200, options [mss 1460,sackOK,TS val 31091970 ecr 0,nop,wscale 7,exp-tfo cookie 9a8e5a15f1deab96], length 75
0x0000: 4500 0093 ce7e 4000 4006 913c 0a02 0564 E....~@.@..<...d
0x0010: 4a7d 80c7 9efb 0050 d78a a37c 0000 0000 J}.....P...|....
0x0020: d002 7210 bb34 0000 0204 05b4 0402 080a ..r..4..........
0x0030: 01da 6d02 0000 0000 0103 0307 fe0c f989 ..m.............
0x0040: 9a8e 5a15 f1de ab96 4845 4144 202f 7572 ..Z.....HEAD./ur
0x0050: 6c20 4854 5450 2f31 2e30 0d0a 486f 7374 l.HTTP/1.0..Host
0x0060: 3a20 7777 772e 676f 6f67 6c65 2e63 6f6d :.www.google.com
0x0070: 2e68 6b0d 0a55 7365 722d 4167 656e 743a .hk..User-Agent:
0x0080: 2048 5454 5069 6e67 2076 322e 332e 330d .HTTPing.v2.3.3.
0x0090: 0a0d 0a ...
20:40:15.295644 IP (tos 0x0, ttl 30, id 42640, offset 0, flags [none], proto TCP (6), length 52)
74.125.128.199.http > 10.2.5.100.40699: Flags [S.], cksum 0x71c1 (correct), seq 1878126810, ack 3616187261, win 42900, options [mss 1430,nop,nop,sackOK,nop,wscale 6], length 0
0x0000: 4500 0034 a690 0000 1e06 1b8a 4a7d 80c7 E..4........J}..
0x0010: 0a02 0564 0050 9efb 6ff1 f0da d78a a37d ...d.P..o......}
0x0020: 8012 a794 71c1 0000 0204 0596 0101 0402 ....q...........
0x0030: 0103 0306 ....
20:40:15.295694 IP (tos 0x0, ttl 64, id 52863, offset 0, flags [DF], proto TCP (6), length 115)
10.2.5.100.40699 > 74.125.128.199.http: Flags [P.], cksum 0x5bf7 (correct), seq 1:76, ack 1, win 229, length 75
0x0000: 4500 0073 ce7f 4000 4006 915b 0a02 0564 E..s..@.@..[...d
0x0010: 4a7d 80c7 9efb 0050 d78a a37d 6ff1 f0db J}.....P...}o...
0x0020: 5018 00e5 5bf7 0000 4845 4144 202f 7572 P...[...HEAD./ur
0x0030: 6c20 4854 5450 2f31 2e30 0d0a 486f 7374 l.HTTP/1.0..Host
0x0040: 3a20 7777 772e 676f 6f67 6c65 2e63 6f6d :.www.google.com
0x0050: 2e68 6b0d 0a55 7365 722d 4167 656e 743a .hk..User-Agent:
0x0060: 2048 5454 5069 6e67 2076 322e 332e 330d .HTTPing.v2.3.3.
0x0070: 0a0d 0a ...
20:40:15.560807 IP (tos 0x0, ttl 30, id 42641, offset 0, flags [none], proto TCP (6), length 40)
74.125.128.199.http > 10.2.5.100.40699: Flags [.], cksum 0x5720 (correct), seq 1, ack 76, win 670, length 0
0x0000: 4500 0028 a691 0000 1e06 1b95 4a7d 80c7 E..(........J}..
0x0010: 0a02 0564 0050 9efb 6ff1 f0db d78a a3c8 ...d.P..o.......
0x0020: 5010 029e 5720 0000 0000 0000 0000 P...W.........
20:40:15.568068 IP (tos 0x0, ttl 30, id 42642, offset 0, flags [none], proto TCP (6), length 269)
74.125.128.199.http > 10.2.5.100.40699: Flags [P.], cksum 0x85ae (correct), seq 1:230, ack 76, win 670, length 229
0x0000: 4500 010d a692 0000 1e06 1aaf 4a7d 80c7 E...........J}..
0x0010: 0a02 0564 0050 9efb 6ff1 f0db d78a a3c8 ...d.P..o.......
0x0020: 5018 029e 85ae 0000 4854 5450 2f31 2e30 P.......HTTP/1.0
0x0030: 2034 3034 204e 6f74 2046 6f75 6e64 0d0a .404.Not.Found..
0x0040: 436f 6e74 656e 742d 5479 7065 3a20 7465 Content-Type:.te
0x0050: 7874 2f68 746d 6c3b 2063 6861 7273 6574 xt/html;.charset
0x0060: 3d55 5446 2d38 0d0a 4461 7465 3a20 5765 =UTF-8..Date:.We
0x0070: 642c 2031 3620 4170 7220 3230 3134 2031 d,.16.Apr.2014.1
0x0080: 323a 3430 3a31 3520 474d 540d 0a53 6572 2:40:15.GMT..Ser
0x0090: 7665 723a 2067 7773 0d0a 436f 6e74 656e ver:.gws..Conten
0x00a0: 742d 4c65 6e67 7468 3a20 3134 3238 0d0a t-Length:.1428..
0x00b0: 582d 5853 532d 5072 6f74 6563 7469 6f6e X-XSS-Protection
0x00c0: 3a20 313b 206d 6f64 653d 626c 6f63 6b0d :.1;.mode=block.
0x00d0: 0a58 2d46 7261 6d65 2d4f 7074 696f 6e73 .X-Frame-Options
0x00e0: 3a20 5341 4d45 4f52 4947 494e 0d0a 416c :.SAMEORIGIN..Al
0x00f0: 7465 726e 6174 652d 5072 6f74 6f63 6f6c ternate-Protocol
0x0100: 3a20 3830 3a71 7569 630d 0a0d 0a :.80:quic....
</code></pre>
</div>
<p>没错,在第一个 SYN 包的时候就把 HEAD 请求带过去了。</p>
<p>但是发现比较奇怪的是很多时候一模一样的命令,SYN 包上就没带数据。</p>
<p>按我的想法,既然还是第一个 SYN 包,客户端这边压根不知道服务器端的情况,那么应该不管服务器端如何 SYN 里都带有 HEAD 请求啊?</p>
<hr />
<p>另外,用 <code class="highlighter-rouge">httping -F</code> 命令测试自己编译的 nginx 的时候,一直都没看到正确的抓包结果,HEAD 请求一直都是在三次握手后发送的。</p>
<p>试图用 systemtap 来追踪一些问题。</p>
<h3 id="nginx--socket--fastopen">第一步确认我的 nginx 的 socket 是不是真的开了 fastopen:</h3>
<p>一个终端运行如下命令:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>stap -e 'probe kernel.function("do_tcp_setsockopt") {printf("%d\n", $optname)}'
</code></pre>
</div>
<p>另一个终端启动nginx,看到前一个终端输出结果为<code class="highlighter-rouge">23</code>,查 <code class="highlighter-rouge">tcp.h</code> 可以看到 23 正是 <code class="highlighter-rouge">TCP_FASTOPEN</code> 没错!</p>
<h3 id="httping--fastopen">第二步确认 httping 发送的时候是不是开了 fastopen:</h3>
<p>一个终端运行如下命令:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>stap -e 'probe kernel.function("tcp_sendmsg") {printf("%d %x\n",$msg->msg_namelen,$msg->msg_flags)}'
</code></pre>
</div>
<p>另一个终端运行最开始提到的 <code class="highlighter-rouge">httping -F</code> 命令,看到前一个终端输出结果为 <code class="highlighter-rouge">16 20000040</code>,查 <code class="highlighter-rouge">socket.h</code> 可以看到 <code class="highlighter-rouge">MSG_FASTOPEN</code> 是 <code class="highlighter-rouge">0x20000000</code>,<code class="highlighter-rouge">MSG_DONTWAIT</code> 是 <code class="highlighter-rouge">0x40</code>,也就是说 httping 也没问题。</p>
<p>现在比较郁闷的一点是:在 <code class="highlighter-rouge">net/ipv4/tcp.c</code> 里,<code class="highlighter-rouge">tcp_sendmsg()</code> 函数会判断 <code class="highlighter-rouge">if ((flags & MSG_FASTOPEN))</code>,就调用 <code class="highlighter-rouge">tcp_sendmsg_fastopen()</code> 函数来处理。但是试图用 systemtap 来调查这个函数的时候,会报一个错:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>WARNING: probe kernel.function("tcp_sendmsg_fastopen@net/ipv4/tcp.c:1005") (address 0xffffffff815cca08) registration error (rc -22)
</code></pre>
</div>
<p>原因还未知。</p>
<p>留记,继续研究。</p>
<hr />
<p>注1:发现 chrome 即使在 <code class="highlighter-rouge">about:flags</code> 里启用了 fastopen 好像也不行,必须命令行 <code class="highlighter-rouge">google-chrome --enable-tcp-fastopen</code> 这样打开才行。</p>
<p>注2:网上看到有人写server和client的demo演示fastopen,但其实不对,demo代码里print的数据是正常三次握手以后socket收到的。这点开tcpdump才能确认到底是什么时候发送的数据。</p>
<hr />
<p><strong>2014 年 04 月 18 日更新:</strong></p>
<p>今天改用 wireshark 看了一下数据包,在第一个 SYN 包没有带请求数据的时候,其实最末尾可选项里是有 fastopen 的,截图如下。看来还是服务器端的问题。下一步研究 <code class="highlighter-rouge">tcp_recvmsg()</code> 函数去。</p>
<p><img src="/images/uploads/wireshark-fastopen.png" alt="" /></p>
Larry Wall 来中国参加 OSTC 和 PerlChina Workshop
2014-04-07T00:00:00+08:00
perl
http://chenlinux.com/2014/04/07/perlchina-mini-workshop-with-larry-wall
<p>见到教主真身真的很让人兴奋。在 OSTC 会场外的茶座抓住机会完成了签名跟合影。</p>
<p>书是从同事那搜刮来的大骆驼,自己的 Perl 书不好意思拿,因为不是教主亲著,不过后来发现绝大多数人都没大骆驼……</p>
<p>穿上了 PerlChina Workshop 2013 的 T恤,教主夫人帮忙在后面扯直了也让教主签名了。</p>
<p>OSTC 上教主讲的是自己跟开源社区的联系和小故事,以他自己最早期的时候的一个小程序 rn (read news) 的开发过程做了示例。</p>
<p>接着一星期后又单独举办了 PerlChina 的 Workshop,场地是一家叫 Happylatte 的手游公司的作坊,很有氛围。作坊环境的图片大家可以进 Linux Deepin 的王勇写的文章里面去看,他拍了超多图片:<a href="http://planet.linuxdeepin.com/archives/5688">http://planet.linuxdeepin.com/archives/5688</a></p>
<p>PerlChina 送给教主一本全员签名的新华大字典,一个 3D 打印的教主头像。</p>
<p><img src="/images/uploads/mysign.jpg" alt="我在字典上签名" /></p>
<p>Linux Deepin的王勇从武汉过来送给教主一个新的笔记本电脑,教主自己那台太老了……</p>
<p><img src="/images/uploads/deepin.jpg" alt="深度的帅小伙" /></p>
<p>教主首先分享,讲述了一些 Perl 语言的设计思想,跟其他语言的思想上的对比。然后现场演示了一个 Perl6 写的小程序,分别用 MoarVM、JVM 和 Parrot 三种虚拟机上的 Rakudo 实现跑了一下给我们看效果。然后基础语法什么的。</p>
<p>时不时还切换到 Perl6 的 IRC 频道上给外国朋友打个招呼,跑个单行命令让 rakudorobot 自动返回结果什么的:</p>
<p><img src="/images/uploads/sayhi.jpg" alt="irc" /></p>
<p>为了演示 Unicode 支持,教主还联系中文环境,直接从字库里搜索了”<img src="/images/uploads/long.png" alt="从一个龙到四个龙,不过后面几个都是Unicode扩展字库里的字,UTF8都不支持,我好不容易找到却没法通过jekyll build编译,只好截图了" />“出来,然后问:为什么没有五个龙叠在一起的字呢?哈哈,看来他是把汉字当做纯象形文字来学习了。于是就少不了著名的”biangbiang面”啦:</p>
<p><img src="/images/uploads/biang.jpg" alt="" /></p>
<p>最后教主也稍微回答了几个我们提前准备的疑问,其中一个是我问的关于 Perl6 是否会去支持直接开发安卓应用的问题,因为有 JVM 实现了嘛。教主意思是”是的,理论上可以。不过实际上现在你要是写肯定会有问题跑不起来的,留作未来吧。”另一个大家都很关心的问题是核心库的问题,一来是 Perl5 的核心库比起 Python 来说少很多,二来是 Perl6 的 Rakudo Star 也要面临这这个第三方模块打包问题了,大家都想知道核心库是怎么选择的,为什么只选择这么多,未来 Perl6 会怎么选?不过教主回答说,核心库这个概念就不该有。语言设计和开发者做好核心,第三方库是发行版的打包者去选择的事情。回答很出乎意料之外,不过想想教主对 Perl6 只写启示录,留给别人做出多种实现,思路似乎是一脉相承的吧。</p>
<p>接着是我们几个人的小分享,本着活泼有趣的原则,都没有讲什么严肃的话题。我讲的是如何操作微博的 API。</p>
<p>最后的互动,教主让大家都说说自己是怎么开始写 Perl 的。一圈说下来起因还是蛮多的。</p>
<p>然后又是签名合影环节。不过这次我就没再去凑热闹了,教主很口耐的估计学每个合影的人的动作搞”镜像”~哈哈</p>
<p><img src="/images/uploads/signtool.jpg" alt="签名必备的印章" /></p>
<p>最后照全家福,大家一起说好不喊茄子喊Wall~~</p>
<p><img src="/images/uploads/quanjiafu.jpg" alt="all" /></p>
腾讯云技术沙龙笔记
2014-03-30T00:00:00+08:00
cloud
nginx
hadoop
http://chenlinux.com/2014/03/30/qcloud-tech
<p>昨天去车库咖啡听了 InfoQ 办的腾讯云图技术沙龙,今天又听了 CSDN 办的开源技术大会上腾讯云的宣讲(没错,就是那个发明了”内部开源”概念的意思),总的来说,幸亏去了昨天的!</p>
<p>沙龙包括三个主题:</p>
<h1 id="section">手机推送服务</h1>
<p>手机推送其实是一个很难有亮点的服务,我之前试用过免费的 JPush 极光推送服务,应该说大家都差不多——引用SDK,通过 RESTful 接口或者网页后台发布通知。</p>
<p>从业务上说,腾讯云提出一个精准投放的推送概念。<br />
这其实跟后面的多维度数据是联系在一起的,腾讯因为本身(可怕)的数据收集能力,可以很容易的区分几个基础维度——年龄、性别、地域。<br />
(今天午饭跟<a href="http://weibo.com/turingbook">@刘江总编</a>在一起,他谈到CSDN如何跟技术社区、出版社一起做技术书籍时,提到类似问题,CSDN 上也有千万级的用户,但是怎么高质量的做推荐才不透支信誉或者徒劳无功呢?)</p>
<p>不过在技术周边介绍中,还是聊到了腾讯的 L5 里的技术点,在这记录一下:</p>
<p>起因是说到<strong>服务扩容,新服务器上线时会自动根据响应质量动态调整其在集群中的权重</strong>。</p>
<p>这里我跟<a href="http://weibo.com/liucy1983">@liu点cy</a>、<a href="http://weibo.com/opendoc">@守住每一天</a>先后猜测并推论了几种在 Nginx 的 upstream 上的实现方式及相关技术。</p>
<ul>
<li><a href="https://github.com/yzprofile/ngx_http_dyups_module">ngx_dyups_module</a></li>
<li><a href="https://github.com/agentzh/lua-upstream-nginx-module">ngx_lua_upstream_module</a></li>
<li><a href="http://www.centurylinklabs.com/auto-loadbalancing-with-fig-haproxy-and-serf/">Serf + Shell + Haproxy</a></li>
</ul>
<p>不过这几种方案一般常见的用途都是上下线而不是权重调整(另一个需要注意的就是在线修改upstream不会同步到nginx.conf文本文件里)。</p>
<p>那么就涉及到下一步问题:<strong>怎么评定响应质量</strong>?</p>
<p>Nginx 里是有个 <a href="https://github.com/cep21/healthcheck_nginx_upstreams">HealthCheck</a> 模块,不过还很基础。<br />
于是联想到 LVS 项目中的调度算法,常见的RR、LC、LBLC和LBLCR,少见的还有NQ、SED。这都算是根据 RS 的情况智能调整流量导向。</p>
<p>后来跟讲师交流,稍微了解到了 L5 内部的一点信息。</p>
<ol>
<li>流量到应用服务之前会经过两层调度(暂称为DNS agent和local agent);</li>
<li>DNS agent 负责多个 local agent 之间的流量调度;</li>
<li>local agent 只负责本组(原话是本机)的应用服务的流量权重调整;</li>
<li>一个新服务器上线,首先要经过一次镜像流量的试运行,达到5个9后才正式上线;</li>
<li>local agent将收到的每秒10万个请求分配 1% 给新服务器,根据平均响应延时和成功率,判定是否合格,合格就继续加流量;</li>
<li>如果某个服务器被判定不合格了,比如低于5个9了,也并不是直接剔除,而是减流量;除非直接成功率只有85%这样,那就是直接踢。</li>
</ol>
<p>从流程里”本机”还是”本组”的用词,很容易让我联想到类似 docker 或者说 PAAS 平台的做法。<br />
我个人猜测确实有可能就是一组服务器,但是同时也是在一台真实主机上的多个容器。</p>
<p><em>这种做法应该适合业务运维尝试;CDN 方面,upstream 列表每次变动都会带来巨大的回源压力,反而是越少变动越好</em></p>
<h1 id="section-1">多维度数据分析</h1>
<p>前面提到了腾讯数据分析上最常用的几个维度就是年龄、性别和地域。但其实做数据挖掘维度是超级多的,讲师举了不少例子。</p>
<p>从腾讯云的概念上来说,这个数据分析主要是几个层次。</p>
<ol>
<li>基础的经过整理和运算得到的 TopView。这个应该就是 Hive 里的表,按照讲师所说,TopView 里有 30 个左右的维度。<br />
从交流来看,这个 Hive 表内容应该就是以 QQ 号为中心的用户行为数据。每天从原始数据里花点时间更新这个表。</li>
<li>选取需要的维度信息做 RollUp。也就是从 TopView 的30个维度数据中选取几个维度做统计分析。这个就是排列组合问题,挨个硬算了。</li>
<li>合作用户如果有自定义维度,并且勾选这个维度做统计分析,就要先退回到计算 TopView 这步,把自定义维度按照 TopView 的处理方式来做。</li>
</ol>
<p>因为对 Hadoop 的 Map/Reduce 稍有了解,也用过 Hive,所以这里的东西不算太难理解。<br />
其实整个重点是在如何用用户行为日志整理得到 TopView 这块,从讲师透露信息看,全腾讯的日志提前清洗过滤到一天只有几个 TB ,不到一百台的小集群几个小时就可以完成全部分析任务。但是这块属于纯 coding 问题,没什么太多可讲的。</p>
<p>在边听演讲的时候我也边思考了一下如果这个问题用 Elasticsearch 做,会怎么样?</p>
<p>由于ES不需要定义 schema,所以类似 TopView 整理这段应该更轻松一些;<br />
RollUp 计算就是写 <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html">bool query</a>。<br />
这个效率如何我不太了解。</p>
<p>(今天的会场上有介绍腾讯大数据平台的,应该跟这个多维度分析不是一个平台,今天的讲师说到他们的平台除了Hadoop这套还用到了pgsql)</p>
<h1 id="section-2">移动动态加速</h1>
<p>这一部分是个人比较关心的部分。移动来源占比越来越大,移动网络质量却一如既往的复杂和烂。如何有效提高移动访问质量现在也是大家都关心的问题,本周网宿也刚发布了他们的私有协议加速产品。</p>
<p>腾讯的做法是也提供了 SDK,但本质上没有做完全的私有协议优化而是尽量利用可靠的自建私有网络,软件的部分应该是今天宣布开源了,地址在:<a href="https://code.csdn.net/Tencent/mna">https://code.csdn.net/Tencent/mna</a>。</p>
<p>SDK 的主要工作流程如下:</p>
<ol>
<li>APP 初次运行,正常访问流程的同时,调用 SDK 开始运作;</li>
<li>SDK 内置有 3 个主要运营商一共 9 个默认 ANS(应该是 application name service 的意思吧)的 IP 地址,同时向这 9 个地址发送 HTTP 请求;<br />
请求内容包括应用使用的域名、 SDK 获取到的本机 IP 和接入运营商(后二者如果获取不到,其实 ANS 通过 HTTP 本身也没问题);</li>
<li>ANS 根据请求,返回尽量近的 OC、RS 和 TEST 三个 IP 地址信息;</li>
<li>SDK 根据最快返回的那个 ANS 的响应结果,开始并发测试本机到 OC 和 TEST 地址的链路情况;<br />
其中,OC 应该是跟 SDK 地址在同省同运营商,并且是负载最低的;TEST 应该是跟 RS 在同机房,作为 RS 的替身来参加链路测试工作;</li>
<li>如果 TEST 测试结果占优,那 APP 继续直连 RS,走正常访问流程就可以了;<br />
如果 OC 测试结果占优,那么 APP 之后的请求,将改为发往 OC 的地址,由 OC 转发给 RS;</li>
<li>在 APP 运行过程中,链路测试是定时每十分钟做一次;当然类似推送这样的长连接服务,不会因为链路测试结果切换而被主动断开。</li>
</ol>
<p>OC 方面的主要工作包括:</p>
<h3 id="tcp-">TCP 代理</h3>
<ul>
<li>TCP 代理就是 sock5 代理。不过针对移动环境做了一些优化,去除了sock5里的一些验证算法;</li>
<li>在 TCP 方面,去掉了 nagle 算法,也就是打开了 TCP_NODELAY 参数。<br />
nagle 算法本身是做小包合大包,提高传输效率的;不过在移动环境下,某个包的丢失或者延迟是个很常态的情况,而 nagle 算法中一个包延迟,所有包都要等在后面的情况就会被放大了,所以打开 TCP_NODELAY 应该可以避免这个情况(个人尚未测试验证过,或许可以相信腾讯)。</li>
</ul>
<h3 id="http-">HTTP 代理</h3>
<p>没细说,应该就是 squid 或者 nginx 之类的。</p>
<h3 id="section-3">集群层面</h3>
<p>每个机房都做了集群,通过 VIP 统一发布。这方面跟<a href="http://weibo.com/opendoc">@守住每一天</a>浅聊了一下通过 MPLS 协议实现 Anycast 来在多机房间维护统一的 VIP。不过看起来大家系统运维跟精通 BGP 的网络专家联系都比较远,这方面还处于有所耳闻的状态。</p>
<p>最后还有一个小问题,就是上面我们看到过好几处,提到”并发”、”同时”这样的字眼,于是当时产生一个疑问:<em>“三个演讲中,都反复强调为了手机省电我们做了这做了那的,为什么为了优化级别的测试工作,却这么频繁和高密度的做并发请求呢?比如 ANS 请求,我只给本运营商的2个ip发请求也可以接受啊?”</em></p>
<p>这个问题正好被旁边围观的另一位听众解答了:手机内的 3G 通信模块,一次大批量的数据发送跟几次小批量的数据发送相比其实更省电。</p>
<p>讲师则从实际效果角度证明,目前的频率和策略,从使用上看,确实看不出来对电量的影响。</p>
Perl5 的 Source Filter 功能
2014-03-10T00:00:00+08:00
perl
http://chenlinux.com/2014/03/10/source-filter-in-perl5
<p>去年在 <a href="https://github.com/stevan/p5-mop-redux">p5-mop-redux</a> 项目里看到他们在 Perl5 里实现了 Perl6 的面向对象设计的很多想法,尤其下面这段示例让人印象深刻:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nv">mop</span><span class="p">;</span>
<span class="nv">class</span> <span class="nv">Point</span> <span class="p">{</span>
<span class="nv">has</span> <span class="nv">$</span><span class="err">!</span><span class="nv">x</span> <span class="nv">is</span> <span class="nv">ro</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="nv">has</span> <span class="nv">$</span><span class="err">!</span><span class="nv">y</span> <span class="nv">is</span> <span class="nv">ro</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="nv">method</span> <span class="nv">clear</span> <span class="p">{</span>
<span class="p">(</span><span class="nv">$</span><span class="err">!</span><span class="nv">x</span><span class="p">,</span> <span class="nv">$</span><span class="err">!</span><span class="nv">y</span><span class="p">)</span> <span class="o">=</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nv">class</span> <span class="nv">Point3D</span> <span class="nv">extends</span> <span class="nv">Point</span> <span class="p">{</span>
<span class="nv">has</span> <span class="nv">$</span><span class="err">!</span><span class="nv">z</span> <span class="nv">is</span> <span class="nv">ro</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="nv">method</span> <span class="nv">clear</span> <span class="p">{</span>
<span class="nv">$self</span><span class="o">-></span><span class="k">next</span><span class="o">::</span><span class="nv">method</span><span class="p">;</span>
<span class="nv">$</span><span class="err">!</span><span class="nv">z</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">my</span> <span class="nv">$p</span> <span class="o">=</span> <span class="nv">Point3D</span><span class="o">-></span><span class="k">new</span><span class="p">(</span><span class="nv">x</span> <span class="o">=></span> <span class="mi">4</span><span class="p">,</span> <span class="nv">y</span> <span class="o">=></span> <span class="mi">2</span><span class="p">,</span> <span class="nv">z</span> <span class="o">=></span> <span class="mi">8</span><span class="p">);</span>
<span class="nb">printf</span><span class="p">(</span><span class="s">"x: %d, y: %d, z: %d\n"</span><span class="p">,</span> <span class="nv">$p</span><span class="o">-></span><span class="nv">x</span><span class="p">,</span> <span class="nv">$p</span><span class="o">-></span><span class="nv">y</span><span class="p">,</span> <span class="nv">$p</span><span class="o">-></span><span class="nv">z</span><span class="p">);</span>
</code></pre>
</div>
<p>这种 <code class="highlighter-rouge">$!x</code> 的变量是怎么实现的?最近几天,又在 CPAN 上看到另一个模块叫 <a href="https://metacpan.org/pod/Perl6::Attributes">Perl6::Attributes</a>,实现了类似的语法。于是点进去一看,实现原来如此简单!</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nb">package</span> <span class="nn">Perl6::</span><span class="nv">Attributes</span><span class="p">;</span>
<span class="k">use</span> <span class="mf">5.006001</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="nb">no</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">our</span> <span class="nv">$VERSION</span> <span class="o">=</span> <span class="s">'0.04'</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Filter::</span><span class="nv">Simple</span> <span class="k">sub </span><span class="p">{</span>
<span class="sr">s/([\$@%&])\.(\w+)/
$1 eq '$' ? "\$self->{'$2'}" : "$1\{\$self->{'$2'}\}"/ge</span><span class="p">;</span>
<span class="sr">s[\./(\w+)][\$self->$1]g</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<p>原来这里用到了 Perl5.7.1 以后提供的一个新特性,叫做 <a href="https://metacpan.org/pod/distribution/Filter/perlfilter.pod">Source Filters</a> 。在解释器把 file 变成 parser 的时候加一层 filter。</p>
Docker Meetup 参会总结
2014-03-09T00:00:00+08:00
docker
puppet
linux
http://chenlinux.com/2014/03/09/thoughts-after-docker-meetup
<p>昨天去车库咖啡参加了 Docker Meetup,一共有三位做了分享。</p>
<p>第一位主要演示用法,这个基本都了解;<br />
第二位描述了一下相关生态圈,我自认算是对DevOps工具和动态了解比较多的人了,听完后对这位自称10年前作为运维的Rails开发者不得不说个佩服,知道的真广泛;<br />
第三位是BAE的技术负责人,很诚恳的介绍了自己是怎么从一抹黑的环境开始摸索着搞 PAAS 平台的,波折的选型中一些想法和顾虑也都很坦白。</p>
<p>问答聊天过程中,大家主要纠结两个疑难:</p>
<ol>
<li>docker 和 puppet 会是什么关系?</li>
<li>docker 和 kvm 会是什么关系?</li>
</ol>
<p>这里我个人也稍微写几句我的想法:</p>
<h1 id="docker--puppet">docker 和 puppet</h1>
<p>docker 无疑是一种非常干净的大规模部署方案。而 puppet 本质是一个配置管理工具(官网说法是通过简洁易懂的DSL描述服务器配置),注意:<strong>这里并没有提到是大规模部署</strong>,事实上 puppet 自己就有好几种完全不同架构设计的部署运行方式。</p>
<p>所以,从概念定义上来说,我不觉得这两者会是一个替代关系。</p>
<p>那么,puppet 目前的用法,如何跟 docker 一起工作呢?从当前技术点上来说有两个不适应:</p>
<ol>
<li>
<p>puppet 非常强大的一件事情是 template 系统和 Facts 变量配合达到的灵活性。但是<strong>在 docker 容器里,Facts 变量是不可信的!</strong><br />
刚才测试了一下,以 <code class="highlighter-rouge">docker -m 56m run ubuntu facter | grep memorysize</code> 得到的结果是主机原始大小512m。所以,我们原先习惯的通过 Facts 变量来自动生成最佳配置的方法失效了。<br />
事实上, docker 官博上关于 metrics 的获取有好几篇文章,也都很明确是从主机上来获取而不是容器内部。</p>
</li>
<li>
<p>puppet 的通用运行方式,是 agent 和 master 通过 SSL 加密交互,根据 agent 的 hostname 来查询对应配置。但是目前的 docker 里,hostname 设置(<code class="highlighter-rouge">docker run -h</code> 参数)是只对容器内部生效的,在容器外部显然无法通过 DNS 反查。<br />
以 docker 的愿景,一台主机上就应该运行几百个容器,在某个 master 里维护 hosts 列表显然不现实。<br />
而且从目前看, docker 对容器间更偏向采用 IP 的方式。比如 <code class="highlighter-rouge">-link</code> 设置的主机,就是在环境变量里提供对方主机 IP。</p>
</li>
</ol>
<p>这两个问题可能更多的不是从技术方面来追求解决它,而是在用法上规避它或者说无视它。</p>
<p>首先,要习惯横向扩展而不是单机提升。<br />
应用压力上来了,第一反应不是“申请提高容器的 memory 限额”这样,而是“再开两个完全一样的容器加入负载均衡”。这就是 fip 工具提供 <code class="highlighter-rouge">fip scale web=2</code>这种命令的场景吧。<br />
这样就规避了 Facts 变量的问题,反正你只会有一种系统一种配置文件,压根用不上异构和模板技术。</p>
<p>其次,从 Vagrant 的 provision 里学用法。<br />
目前 Dockerfile 的 <code class="highlighter-rouge">RUN</code> 指令其实很类似 Vagrant 的 provision 中的 shell 实现。而 Vagrant 的 provision 实现还包括 puppet、chef等等。所以我们或许能琢磨一种替代 RUN 的优雅的 docker 镜像构建方式。<br />
比如 <a href="http://librarian-puppet.com/">puppet-librarian</a> 的做法或许就是一个思路。Dockerfile 里 只需要 <code class="highlighter-rouge">ADD</code> 一个 Puppetfile,然后 <code class="highlighter-rouge">RUN</code> 一个 librarian-puppet 命令完成容器内一切配置。</p>
<h1 id="docker--kvm">docker 和 kvm</h1>
<p>前面提到了 docker 中系统性能数据的采集问题。这或许就是容器和虚拟化一个差别问题,即便未来大家越来越普遍采购 ops 产品而不是自己搭建监控系统,也不会完全放心的认可主机提供商的系统性能数据,至少也还有一个核算和度量问题。</p>
<p>此外,容器目前比较普遍的一个用法,是一个容器里只跑一个业务进程。一个完整的业务系统的每个部分,都通过分散的各种服务相互走 API 来调用。迁移到这种环境,对传统业务显然是有重构压力的。而 kvm 虚拟机则基本没有这个问题。<br />
当然,最近也已经看到文章在讨论单个 docker 容器里运行多个不同业务进程的问题。这方面,如果 docker 真有心往替代 kvm 努力,除了网络方面的硬技术外,这个 PAAS 层已经养成的思维逻辑也需要改变。</p>
<p>OK,说到网络问题。目前 docker 的运用,通过 <code class="highlighter-rouge">-link</code> 来连接,或者通过 etcd、serf 这类工具来获取想要连接的其他服务器的 IP,都是一种在相同主机上的应用。<br />
看 <code class="highlighter-rouge">pipework</code> 和相关文章,似乎 <code class="highlighter-rouge">openswitch</code> 也只是做单个宿主机之上的 VLAN 划分管理? SDN 到底是怎么回事,我现在还完全不了解。</p>
<p>PAAS 层的另一个习惯用法,在第三个演讲中也提到,就是一般对程序的任何更新,都是重新创建一个新容器,然后在中控转发里转移流量导向,然后删除原有容器。这个和现有 kvm 云主机的玩法也是不一样的。<br />
现在还不好评价哪种做法更优。不过个人有个疑惑: BAE 既然试图做到像 kvm 虚拟机一样,对一个用户长期锁定一个 docker 容器使用而不是随着更新开关新容器,那么整个平台上容器的创建删除频率就大大降低了,针对每个用户,整个生命周期里就只有一次初创建,那么他们为什么同时又还在纠结于容器创建和删除的速度太慢,要在 5s 内完成呢?</p>
<p><strong>附</strong></p>
<p>提到的从 warden 学来的 wsh 听起来蛮有趣~</p>
如何搜索 Elasticsearch 中存储的动态请求 URL
2014-03-07T00:00:00+08:00
logstash
elasticsearch
http://chenlinux.com/2014/03/07/howto-search-dynamic-url-in-elasticsearch
<p>当我们用 logstash 处理 WEB 服务器访问日志的时候,肯定就涉及到一个后期查询的问题。</p>
<p>可能一般我们在 Kibana 上更多的是针对响应时间做数值统计,针对来源IP、域名或者客户端情况做分组统计。但是如果碰到这么个问题的时候呢——<strong>过滤所有动态请求的响应时间</strong>。</p>
<p>这时候你可能会发现一个问题:我们肯定都是用 URL 里带有问号? 来作为过滤条件。但是实际是 Kibana 里一条数据都过滤不出来。</p>
<p>于是我开测试库模拟了一下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># 插入两条数据</span>
curl http://localhost:9200/test/log/1 -d <span class="s1">'{"url":"http://locahost/index.html"}'</span>
curl http://localhost:9200/test/log/2 -d <span class="s1">'{"url":"http://locahost/index.php?key=value"}'</span>
<span class="c"># 搜索显示全部数据</span>
curl http://localhost:9200/test/log/_search?pretty<span class="o">=</span>1 -d <span class="s1">'{"query":{"regexp":{"url":{"value":".*"}}}}'</span>
<span class="c"># 搜索返回请求格式解析失败</span>
curl http://localhost:9200/test/log/_search?pretty<span class="o">=</span>1 -d <span class="s1">'{"query":{"regexp":{"url":{"value":"\?.*"}}}}'</span>
<span class="c"># 搜索返回空数据</span>
curl http://localhost:9200/test/log/_search?pretty<span class="o">=</span>1 -d <span class="s1">'{"query":{"regexp":{"url":{"value":".*\\?.*"}}}}'</span>
</code></pre>
</div>
<p>后来发现问题出在分词上面。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># 删除之前的测试数据和索引</span>
curl -XDELETE http://localhost:9200/test/log
<span class="c"># 预定义索引类型的映射,url字段在索引的时候不分词</span>
curl http://localhost:9200/test/log/_mapping -d <span class="s1">'{"log":{"properties":{"url":{"index":"not_analyzed","type":"string"}}}}'</span>
<span class="c"># 还是插入两条数据</span>
curl http://localhost:9200/test/log/1 -d <span class="s1">'{"url":"http://locahost/index.html"}'</span>
curl http://localhost:9200/test/log/2 -d <span class="s1">'{"url":"http://locahost/index.php?key=value"}'</span>
<span class="c"># 同样的搜索请求,返回了一条结果(index.php?这条)</span>
curl http://localhost:9200/test/log/_search?pretty<span class="o">=</span>1 -d <span class="s1">'{"query":{"regexp":{"url":{"value":".*\\?.*"}}}}'</span>
</code></pre>
</div>
<p>上面这个搜索还可以简写成 Query DSL 的样式:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>curl 'http://localhost:9200/test/log/_search?q=url:/.*\\?.*/&pretty=1'
</code></pre>
</div>
<p>而在 Logstash 比较新的 1.3.3 版本之后,有自带的 template 定义,会对每个 fields 采用 <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/_multi_fields.html">multi-fields</a> 特性,也就是除了默认分词的 <code class="highlighter-rouge">URL</code> 字段以外,还有一个 <code class="highlighter-rouge">URL.raw</code> 字段都是不分词的。所以只要过滤这个字段就可以了。</p>
<p>(<em>注意,ES1.0版的multi-fields的template写法完全不一样了,所以要用这个特性的童鞋还是谨慎测试logstash和es的版本配套</em>)</p>
<p>Medcl 大神提示我:<strong>不指定 mapping 的情况下,ES 默认采用 unigram 分词</strong>。也就是切成尽可能小的单词。</p>
转换 diagramo 绘制的拓扑图成 fig.yml 格式
2014-03-07T00:00:00+08:00
docker
perl
php
docker
http://chenlinux.com/2014/03/07/genarate-fig-from-diagramo
<p>前几天在微博上跟 <a href="http://weibo.com/panjunyong">@易度-潘俊勇</a> 在评论里提到,已经有了 <a href="http://orchardup.github.io/fig/">Fig</a> 工具可以通过写一个 <code class="highlighter-rouge">fig.yml</code> 来快速定义主机上各 docker 容器的配置和角色。如果再进一步,可以通过绘图的方式,直接拖拽生成整个 docker 集群,那就更好了。</p>
<blockquote>
<p>这个FIG挺有趣的,我自己写了一个类似的脚本。<br />
不过我觉得终极的解决方案是画个关系图,就配置好了。<br />
这个图的存储形式应该就是这个FIG,或者FIG可以转换为图。然后又可以转换为systemd的配置文件。</p>
</blockquote>
<p>画关系图,桌面上肯定是 visio,visio保存成 XML 后分析 XML 就可以了。不过 visio 本身也算笨重的了,如果可以在浏览器中完成这个工作,才算够 cool!</p>
<p>网页上的 visio 已经有些产品,不过有名的几个都是有限免费试用的。好在找到一个叫做 <a href="http://diagramo.com">diagramo</a> 的项目,虽然提供的元素图表不多,但是也够用了。</p>
<p>下载源码包,在 LAMP/LEMP 环境下就直接跑起来,首次访问会要求注册一个用户名。环境配置中有一点必须重点点出来的:</p>
<p><strong>Apache/Nginx 上配置的 <code class="highlighter-rouge">server_name</code> 必须跟你浏览器访问的完全一致</strong></p>
<p>我曾经因为测试,所以写了个 localhost 做 server_name,然后用服务器 IP 地址来访问页面,结果在绘图完成保存的时候会出错!<strong>因为这是一个 HTML5 项目,保存这步是调用的 <code class="highlighter-rouge">canvas.toDataURL()</code> 函数,这个函数有强制性安全限定,以保证调用这个函数的页面,跟生成的图片路径必须是同一个域名!否则跨域抓图太方便了。</strong></p>
<p>(写到这里感慨一下,chrome的调试工具不会用,这问题最后还是在 IE开发者工具的帮助下发现的 ==!)</p>
<p>然后就可以画关系图了,比如下图这样:</p>
<p><img src="/images/uploads/dia.png" alt="sample of diagramo" /></p>
<p>点击保存后,就会在服务器上的 <code class="highlighter-rouge">$document_root/editor/data/diagrams</code> 目录下生成对应的 <code class="highlighter-rouge">.dia</code> 和 <code class="highlighter-rouge">.png</code> 文件。这个所谓的 <code class="highlighter-rouge">.dia</code> 文件,其实内容就是 JSON数据。下面我们只要抽取 JSON 里有关的数据就可以了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nn">File::</span><span class="nv">Slurp</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">JSON</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">YAML</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Test::Deep::</span><span class="nv">NoTest</span><span class="p">;</span>
<span class="k">use</span> <span class="mf">5.010</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$hash</span> <span class="o">=</span> <span class="nv">from_json</span><span class="p">(</span> <span class="nv">read_file</span><span class="p">(</span> <span class="nv">$ARGV</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="p">)</span> <span class="p">);</span>
<span class="k">my</span> <span class="nv">$hostinfo</span><span class="p">;</span>
<span class="k">for</span> <span class="k">my</span> <span class="nv">$host</span> <span class="p">(</span> <span class="nv">@</span><span class="p">{</span> <span class="nv">$hash</span><span class="o">-></span><span class="p">{</span><span class="nv">s</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">figures</span><span class="p">}</span> <span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$hostinfo</span><span class="o">-></span><span class="p">{</span> <span class="nv">$host</span><span class="o">-></span><span class="p">{</span><span class="nv">id</span><span class="p">}</span> <span class="p">}</span> <span class="o">=</span> <span class="nv">Load</span><span class="p">(</span> <span class="nv">$host</span><span class="o">-></span><span class="p">{</span><span class="nv">primitives</span><span class="p">}</span><span class="o">-></span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">-></span><span class="p">{</span><span class="nv">str</span><span class="p">}</span> <span class="p">);</span>
<span class="p">}</span>
<span class="k">for</span> <span class="k">my</span> <span class="nv">$conn</span> <span class="p">(</span> <span class="nv">@</span><span class="p">{</span> <span class="nv">$hash</span><span class="o">-></span><span class="p">{</span><span class="nv">m</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">connectors</span><span class="p">}</span> <span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$connid</span> <span class="o">=</span> <span class="nv">$conn</span><span class="o">-></span><span class="p">{</span><span class="nv">id</span><span class="p">};</span>
<span class="k">my</span> <span class="nv">$start</span> <span class="o">=</span> <span class="nv">$conn</span><span class="o">-></span><span class="p">{</span><span class="nv">turningPoints</span><span class="p">}</span><span class="o">-></span><span class="p">[</span><span class="mi">0</span><span class="p">];</span>
<span class="k">my</span> <span class="nv">$end</span> <span class="o">=</span> <span class="nv">$conn</span><span class="o">-></span><span class="p">{</span><span class="nv">turningPoints</span><span class="p">}</span><span class="o">-></span><span class="p">[</span><span class="mi">1</span><span class="p">];</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$conn</span><span class="o">-></span><span class="p">{</span><span class="nv">endStyle</span><span class="p">}</span> <span class="ow">eq</span> <span class="s">'Normal'</span> <span class="ow">and</span> <span class="nv">$conn</span><span class="o">-></span><span class="p">{</span><span class="nv">startStyle</span><span class="p">}</span> <span class="ow">eq</span> <span class="s">'Arrow'</span> <span class="p">)</span> <span class="p">{</span>
<span class="p">(</span> <span class="nv">$start</span><span class="p">,</span> <span class="nv">$end</span> <span class="p">)</span> <span class="o">=</span> <span class="p">(</span> <span class="nv">$end</span><span class="p">,</span> <span class="nv">$start</span> <span class="p">);</span>
<span class="p">}</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$startid</span><span class="p">,</span> <span class="nv">$endid</span> <span class="p">);</span>
<span class="k">for</span> <span class="k">my</span> <span class="nv">$point</span> <span class="p">(</span> <span class="nv">@</span><span class="p">{</span> <span class="nv">$hash</span><span class="o">-></span><span class="p">{</span><span class="nv">m</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">connectionPoints</span><span class="p">}</span> <span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">eq_deeply</span><span class="p">(</span> <span class="nv">$point</span><span class="o">-></span><span class="p">{</span><span class="nv">point</span><span class="p">},</span> <span class="nv">$start</span> <span class="p">)</span>
<span class="ow">and</span> <span class="nv">$point</span><span class="o">-></span><span class="p">{</span><span class="nv">parentId</span><span class="p">}</span> <span class="o">!=</span> <span class="nv">$connid</span>
<span class="ow">and</span> <span class="nb">exists</span> <span class="nv">$hostinfo</span><span class="o">-></span><span class="p">{</span> <span class="nv">$point</span><span class="o">-></span><span class="p">{</span><span class="nv">parentId</span><span class="p">}</span> <span class="p">}</span> <span class="p">)</span>
<span class="p">{</span>
<span class="nv">$startid</span> <span class="o">=</span> <span class="nv">$point</span><span class="o">-></span><span class="p">{</span><span class="nv">parentId</span><span class="p">};</span>
<span class="p">}</span>
<span class="k">elsif</span> <span class="p">(</span> <span class="nv">eq_deeply</span><span class="p">(</span> <span class="nv">$point</span><span class="o">-></span><span class="p">{</span><span class="nv">point</span><span class="p">},</span> <span class="nv">$end</span> <span class="p">)</span>
<span class="ow">and</span> <span class="nv">$point</span><span class="o">-></span><span class="p">{</span><span class="nv">parentId</span><span class="p">}</span> <span class="o">!=</span> <span class="nv">$connid</span>
<span class="ow">and</span> <span class="nb">exists</span> <span class="nv">$hostinfo</span><span class="o">-></span><span class="p">{</span> <span class="nv">$point</span><span class="o">-></span><span class="p">{</span><span class="nv">parentId</span><span class="p">}</span> <span class="p">}</span> <span class="p">)</span>
<span class="p">{</span>
<span class="nv">$endid</span> <span class="o">=</span> <span class="nv">$point</span><span class="o">-></span><span class="p">{</span><span class="nv">parentId</span><span class="p">};</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$startname</span><span class="p">)</span> <span class="o">=</span> <span class="nb">keys</span> <span class="nv">%</span><span class="p">{</span> <span class="nv">$hostinfo</span><span class="o">-></span><span class="p">{</span><span class="nv">$startid</span><span class="p">}</span> <span class="p">};</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$endname</span><span class="p">)</span> <span class="o">=</span> <span class="nb">keys</span> <span class="nv">%</span><span class="p">{</span> <span class="nv">$hostinfo</span><span class="o">-></span><span class="p">{</span><span class="nv">$endid</span><span class="p">}</span> <span class="p">};</span>
<span class="nb">push</span> <span class="nv">@</span><span class="p">{</span> <span class="nv">$hostinfo</span><span class="o">-></span><span class="p">{</span><span class="nv">$startid</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">$startname</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nb">link</span><span class="p">}</span> <span class="p">},</span> <span class="nv">$endname</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">say</span> <span class="nv">Dump</span> <span class="p">{</span> <span class="nb">map</span> <span class="p">{</span> <span class="k">my</span> <span class="p">(</span><span class="nv">$k</span><span class="p">)</span> <span class="o">=</span> <span class="nb">keys</span> <span class="nv">$_</span><span class="p">;</span> <span class="nv">$k</span> <span class="o">=></span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="nv">$y</span><span class="p">}</span> <span class="p">}</span> <span class="nb">values</span> <span class="nv">$hostinfo</span><span class="p">};</span>
</code></pre>
</div>
<p>生成的 <code class="highlighter-rouge">fig.yml</code> 如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nn">---</span>
<span class="s">Haproxy</span><span class="pi">:</span>
<span class="s">link</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">Serf</span>
<span class="s">Nginx1</span><span class="pi">:</span>
<span class="s">link</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">Serf</span>
<span class="s">Serf</span><span class="pi">:</span>
<span class="s">Nginx2</span><span class="pi">:</span>
<span class="s">link</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">Serf</span>
<span class="s">MySQL</span><span class="pi">:</span>
<span class="s">link</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">Serf</span>
</code></pre>
</div>
<p>只是根据关系图生成了 link,其他配置都在图里的 Text 里照样写 yaml 格式,会自动带入。当然,示例另一个意思是:大家尽量都只 link 像 serf/etcd 这样的服务自动发现服务器。在 docker 层面就简洁明了。</p>
Gearman 任务的优先级
2014-02-20T00:00:00+08:00
perl
gearman
http://chenlinux.com/2014/02/20/gearman-task-priority
<p>今天同事跟我说 Gearman 客户端添加任务的时候似乎设置优先级没有效果,于是去实现了一下,发现 Gearman 的任务优先级只有在任务本身属性完全一致的时候才能起到作用。比如说:新提交的 background 任务优先级虽然是 high,也不会在已经提交的<em>非</em> background、优先级是 low 的任务之前执行。</p>
<p>考虑到之前没用过优先级,这里贴一下测试代码当做笔记:</p>
<h1 id="worker">worker</h1>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nn">Gearman::XS::</span><span class="nv">Worker</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$worker</span> <span class="o">=</span> <span class="k">new</span> <span class="nn">Gearman::XS::</span><span class="nv">Worker</span><span class="p">;</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$host</span><span class="p">,</span> <span class="nv">$port</span><span class="p">)</span> <span class="o">=</span> <span class="p">(</span><span class="s">'10.4.1.21'</span><span class="p">,</span> <span class="mi">4730</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$ret</span> <span class="o">=</span> <span class="nv">$worker</span><span class="o">-></span><span class="nv">add_server</span><span class="p">(</span><span class="nv">$host</span><span class="p">,</span> <span class="nv">$port</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$ret</span> <span class="o">=</span> <span class="nv">$worker</span><span class="o">-></span><span class="nv">add_function</span><span class="p">(</span><span class="s">"reverse"</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="o">\&</span><span class="nb">reverse</span><span class="p">,</span> <span class="nv">$options</span><span class="p">);</span>
<span class="k">while</span> <span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$ret</span> <span class="o">=</span> <span class="nv">$worker</span><span class="o">-></span><span class="nv">work</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">reverse</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$job</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$workload</span> <span class="o">=</span> <span class="nv">$job</span><span class="o">-></span><span class="nv">workload</span><span class="p">();</span>
<span class="k">my</span> <span class="nv">$result</span> <span class="o">=</span> <span class="nv">$workload</span><span class="p">;</span>
<span class="nb">printf</span><span class="p">(</span><span class="s">"Job=%s Function_Name=%s Workload=%s Result=%s\n"</span><span class="p">,</span>
<span class="nv">$job</span><span class="o">-></span><span class="nv">handle</span><span class="p">(),</span> <span class="nv">$job</span><span class="o">-></span><span class="nv">function_name</span><span class="p">(),</span> <span class="nv">$job</span><span class="o">-></span><span class="nv">workload</span><span class="p">(),</span> <span class="nv">$result</span><span class="p">);</span>
<span class="nb">sleep</span><span class="p">(</span><span class="mi">5</span><span class="p">);</span>
<span class="k">return</span> <span class="nv">$result</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
<h1 id="client">client</h1>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nn">Gearman::XS::</span><span class="nv">Client</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Time::</span><span class="nv">HiRes</span> <span class="sx">qw/time/</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$client</span> <span class="o">=</span> <span class="k">new</span> <span class="nn">Gearman::XS::</span><span class="nv">Client</span><span class="p">;</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$host</span><span class="p">,</span> <span class="nv">$port</span><span class="p">)</span> <span class="o">=</span> <span class="p">(</span><span class="s">'10.4.1.21'</span><span class="p">,</span> <span class="mi">4730</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$ret</span> <span class="o">=</span> <span class="nv">$client</span><span class="o">-></span><span class="nv">add_server</span><span class="p">(</span><span class="nv">$host</span><span class="p">,</span> <span class="nv">$port</span><span class="p">);</span>
<span class="k">while</span> <span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$ret</span><span class="p">,</span> <span class="nv">$job_handle</span><span class="p">)</span> <span class="o">=</span> <span class="nv">$client</span><span class="o">-></span><span class="nv">do_background</span><span class="p">(</span><span class="s">"reverse"</span><span class="p">,</span> <span class="s">'low'</span><span class="o">.</span><span class="nb">time</span><span class="p">()</span> <span class="p">);</span>
<span class="p">}</span>
</code></pre>
</div>
<h1 id="high-client">high-client</h1>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nn">Gearman::XS::</span><span class="nv">Client</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Time::</span><span class="nv">HiRes</span> <span class="sx">qw/time/</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$client</span> <span class="o">=</span> <span class="k">new</span> <span class="nn">Gearman::XS::</span><span class="nv">Client</span><span class="p">;</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$host</span><span class="p">,</span> <span class="nv">$port</span><span class="p">)</span> <span class="o">=</span> <span class="p">(</span><span class="s">'10.4.1.21'</span><span class="p">,</span> <span class="mi">4730</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$ret</span> <span class="o">=</span> <span class="nv">$client</span><span class="o">-></span><span class="nv">add_server</span><span class="p">(</span><span class="nv">$host</span><span class="p">,</span> <span class="nv">$port</span><span class="p">);</span>
<span class="k">while</span> <span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$ret</span><span class="p">,</span> <span class="nv">$job_handle</span><span class="p">)</span> <span class="o">=</span> <span class="nv">$client</span><span class="o">-></span><span class="nv">do_high_background</span><span class="p">(</span><span class="s">"reverse"</span><span class="p">,</span> <span class="s">'high'</span><span class="o">.</span><span class="nb">time</span><span class="p">()</span> <span class="p">);</span>
<span class="p">}</span>
</code></pre>
</div>
<p>同时运行三个脚本,可以看到 worker 的输出,一直都是这样:</p>
<p>Job=H:YZSJHL1-21.opi.com:29434227 Function_Name=reverse Workload=high:1392887687.87583 Result=high:1392887687.87583<br />
Job=H:YZSJHL1-21.opi.com:29434228 Function_Name=reverse Workload=high:1392887687.87594 Result=high:1392887687.87594<br />
Job=H:YZSJHL1-21.opi.com:29434229 Function_Name=reverse Workload=high:1392887687.87605 Result=high:1392887687.87605</p>
<p>全都是高优先级的任务</p>
Facts 变量中 lsbdistid 和 operatingsystem 的区别
2014-02-20T00:00:00+08:00
linux
ruby
http://chenlinux.com/2014/02/20/diff-between-lsddistid-and-operatingsystem-facts-variables
<p>Facts 变量是 puppet 里广泛使用的东西。在多种操作系统的混合环境中,通过 Facts 变量灵活定义不同的 package 名称、file 路径等应该是非常好用的办法。</p>
<p>不过关于操作系统,存在两类 Facts 变量,分别是 <code class="highlighter-rouge">lsbdistid</code> 和 <code class="highlighter-rouge">operatingsystem</code>。一般情况下,这两者结果基本一致,大家(至少我周围是)习惯采用 <code class="highlighter-rouge">operatingsystem</code> 这个一目了然的变量。</p>
<p>但是前两天发现有些机器的 puppet agent 运行失败,debug 后发现,居然是 <code class="highlighter-rouge">operatingsystem</code> 变量匹配不上!这台 CentOS 的服务器的 <code class="highlighter-rouge">operatingsystem</code> 结果是 OracleLinux!</p>
<p>翻看这两个变量的获取代码,他们的获取办法并不一致。</p>
<ul>
<li><code class="highlighter-rouge">lsbdistid</code> 是通过运行 <code class="highlighter-rouge">lsb_release -i -s</code> 命令获取的;</li>
<li><code class="highlighter-rouge">operatingsystem</code> 是通过一串超长的 if-elif-else 逻辑来判断的。恰好其中探测 <code class="highlighter-rouge">/etc/oracle-release</code> 是否存在的步骤优先于探测 <code class="highlighter-rouge">/etc/redhat-release</code> 的步骤。</li>
</ul>
<p>而这台服务器上,不知道怎么被人安装了一个 <code class="highlighter-rouge">oraclelinux-release-5-8.0.2</code> 的软件包,这个包里只有一个文件,就是 <code class="highlighter-rouge">/etc/oracle-release</code>!</p>
<p>这个软件包怎么出现的可以慢慢追查,但是这件事情本身提醒我们,<code class="highlighter-rouge">operatingsystem</code> 变量的获取方式过于简单,这些文本文件稍有问题可能就会导致错误。所以在只有 Linux 类服务器的情况,还是尽量确保所有节点都安装有 <code class="highlighter-rouge">lsb_release</code> 命令然后使用 <code class="highlighter-rouge">lsbdistid</code> 变量吧。</p>
用 logstash 统计 Nginx 的 http_accounting 模块输出
2014-02-19T00:00:00+08:00
logstash
nginx
logstash
syslog
http://chenlinux.com/2014/02/19/ngx-accounting-to-logstash
<p>继续捡宝贝~</p>
<p>http_accounting 是 Nginx 的一个第三方模块,会每隔5分钟自动统计 Nginx 所服务的流量,然后发送给 syslog。</p>
<p>流量以 <code class="highlighter-rouge">accounting_id</code> 为标签来统计,这个标签可以设置在 <code class="highlighter-rouge">server {}</code> 级别,也可以设置在 <code class="highlighter-rouge">location /urlpath {}</code> 级别,非常灵活。<br />
统计的内容包括响应字节数,各种状态码的响应个数。</p>
<p>公司原先是有一套基于 rrd 的<a href="https://github.com/Lax/ngx_http_accounting_module-utils">系统</a>,来收集处理这些 syslog 数据并作出预警判断、异常报警。不过今天不讨论这个,而是试图用最简单的方式,快速的搞定类似的中心平台。</p>
<p>这里当然是 logstash 的最佳用武之地。</p>
<p><code class="highlighter-rouge">logstash.conf</code> 示例如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>input {
syslog {
port => 29124
}
}
filter {
grok {
match => [ "message", "^%{SYSLOGTIMESTAMP:timestamp}\|\| pid:\d+\|from:\d{10}\|to:\d{10}\|accounting_id:%{WORD:accounting}\|requests:%{NUMBER:req:int}\|bytes_out:%{NUMBER:size:int}\|(?:200:%{NUMBER:count.200:int}\|?)?(?:206:%{NUMBER:count.206:int}\|?)?(?:301:%{NUMBER:count.301:int}\|?)?(?:302:%{NUMBER:count.302:int}\|?)?(?:304:%{NUMBER:count.304:int}\|?)?(?:400:%{NUMBER:count.400:int}\|?)?(?:401:%{NUMBER:count.401:int}\|?)?(?:403:%{NUMBER:count.403:int}\|?)?(?:404:%{NUMBER:count.404:int}\|?)?(?:499:%{NUMBER:count.499:int}\|?)?(?:500:%{NUMBER:count.500:int}\|?)?(?:502:%{NUMBER:count.502:int}\|?)?(?:503:%{NUMBER:count.503:int}\|?)?"
}
date {
match => [ "timestamp", "MMM dd HH:mm:ss", "MMM d HH:mm:ss" ]
}
}
output {
elasticsearch {
embedded => true
}
}
</code></pre>
</div>
<p>然后运行 <code class="highlighter-rouge">java -jar logstash-1.3.3-flatjar.jar agent -f logstash.conf</code> 即可完成收集入库!<br />
再运行 <code class="highlighter-rouge">java -jar logstash-1.3.3-flatjar.jar web</code> 即可在9292端口访问到 Kibana 界面。</p>
<p>然后我们开始配置界面成自己需要的样子:</p>
<ol>
<li>Top-N 的流量图</li>
</ol>
<p>点击 Query 搜索栏左边的有色圆点,弹出搜索栏配置框,默认是 <code class="highlighter-rouge">lucene</code> 搜索方式,改成 <code class="highlighter-rouge">topN</code> 搜索方式。然后填入分析字段为 accounting。</p>
<p>点击 Event Over Time 柱状图右上角第二个的 <code class="highlighter-rouge">Configure</code> 小图标,弹出图表配置框:</p>
<ul>
<li>在 <code class="highlighter-rouge">Panel</code> 选项卡中修改 <code class="highlighter-rouge">Chart value</code> 的 <code class="highlighter-rouge">count</code> 为 <code class="highlighter-rouge">total</code>,<code class="highlighter-rouge">Value Field</code> 设置为 size,<strong>勾选 <code class="highlighter-rouge">Seconds</code> 项,转换 size 的累加值成每秒带宽(不然 interval 变化会导致累加值变化)</strong>;</li>
<li>在 <code class="highlighter-rouge">Style</code> 选项卡中修改 <code class="highlighter-rouge">Chart Options</code> 的 <code class="highlighter-rouge">Bars</code> 勾选项为 <code class="highlighter-rouge">Lines</code>,<code class="highlighter-rouge">Y Format</code> 为 bytes;</li>
<li>在 <code class="highlighter-rouge">Queries</code> 选项卡中修改 <code class="highlighter-rouge">Charted Queries</code> 为 <code class="highlighter-rouge">selected</code>,然后点中右侧列出的请求中所需要的那项(当前只有一个,就是<code class="highlighter-rouge">*</code>)。</li>
</ul>
<p>保存退出配置框,即可看到该图表开始自动更新。</p>
<ol>
<li>50x 错误的技术图</li>
</ol>
<p>点击 Query 搜索栏右边的 + 号,添加新的 Query 搜索栏,然后在新搜索栏里输入需要搜索的内容,比如 <code class="highlighter-rouge">count.500</code>;</p>
<p>鼠标移动到流量图最左侧,会移出 Panel 快捷选项,点击最底下的 + 号选项添加新的 Panel:</p>
<ul>
<li>选择 Panel 类型为 <code class="highlighter-rouge">histogram</code>;</li>
<li>选择 Queries 类型为 <code class="highlighter-rouge">selected</code>,然后点中右侧列出的请求中所需要的那项(现在出现两个了,选中我们刚添加的 <code class="highlighter-rouge">count.500</code>)。</li>
</ul>
<p>保存退出,即可看到新多出来一行,左侧三分之一(默认是span4,添加的时候可以选择)的位置有了一个柱状图。</p>
<p>重复这个步骤,添加 502/503 的柱状图。</p>
<ol>
<li>仪表盘设置存档</li>
</ol>
<p>页面右上角选择 <code class="highlighter-rouge">Save</code> 小图标保存即可。之后再上界面后,就可以点击右上角的 <code class="highlighter-rouge">Load</code> 小图标自动加载。</p>
<p><img src="/images/uploads/logstash-ngx-accounting.png" alt="" /></p>
<p>上面这个 grok 写的很难看,不过似乎也没有更好的办法~下一步会研究在这个基础上合并 skyline 预警。</p>
<hr />
<p>2014 年 5 月 10 日更新:</p>
<p>在 <a href="http://logstash.net/docs/1.4.1/">logstash/docs</a> 上发现一个 filter 叫 kv,很适合这个场景,可以大大简化 grok 工作,新的 filter 配置如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>filter {
grok {
match => [ "message", "^%{SYSLOGTIMESTAMP:timestamp}\|\| pid:\d+\|from:\d{10}\|to:\d{10}\|accounting_id:%{WORD:accounting}\|requests:%{NUMBER:req:int}\|bytes_out:%{NUMBER:size:int}\|%{DATA:status}"
}
kv {
target => "code"
source => "status"
field_split => "|"
value_split => ":"
}
ruby {
code => "n={};event['code'].each_pair{|x,y|n[x]=y.to_i};event['code']=n"
}
}
</code></pre>
</div>
<p>不晓得为什么 filter/mutate 不提供转换 Hash 的功能,所以只能把这行写在 filter/ruby 里面。kv 截出来的 value 默认都是字符串类型。</p>
<hr />
<p>2014 年 5 月 28 日更新:</p>
<p>发现默认的 LVS 检查导致的 400 会记录到默认的 accounting 组(“default”)里,虽然不占带宽,却占不少请求数。这类日志可以在 logstash层面就干掉:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>filter {
grok {
match => [ "message", "^%{SYSLOGTIMESTAMP:timestamp}\|\| pid:\d+\|from:\d{10}\|to:\d{10}\|accounting_id:%{WORD:accounting}\|requests:%{NUMBER:req:int}\|bytes_out:%{NUMBER:size:int}\|%{DATA:status}"
}
if [accounting] == 'default' {
drop { }
} else {
kv {
target => "code"
source => "status"
field_split => "|"
value_split => ":"
}
ruby {
code => "n={};event['code'].each_pair{|x,y|n[x]=y.to_i};event['code']=n"
}
}
}
</code></pre>
</div>
<p>另外说明一下,<code class="highlighter-rouge">ngx_http_accounting_module</code> 中设定 <code class="highlighter-rouge">http_accounting_id</code> 这步是预先处理的,所以只能写固定字符串,不能用 <code class="highlighter-rouge">$host</code> 之类的 nginx.conf 变量。</p>
squid-ssd方案和trafficserver的interim层的异同
2014-02-18T00:00:00+08:00
cache
trafficserver
squid
linux
http://chenlinux.com/2014/02/18/diff-between-squid-ssd-and-ats-interim
<p>最近重新捡起来两年前做的 cache 软件测试对比,把原先的 trafficserver 淘宝分支升级到了现在的社区主分支,主要区别就是配置文件里不再直接叫 <code class="highlighter-rouge">ssd.storage</code>,而是正规化的起了一个名字叫<code class="highlighter-rouge">interim cache layer</code>。</p>
<p>运行结果和当初类似,SATA 盘的 ioutil% 依然是远高于鄙司自创的 squid-ssd 方案。</p>
<p>于是沉下心来思考了一下为什么会有这么大的差距。</p>
<p>首先,squid-ssd 的设计其实非常简单,参照 Facebook 的 flashcache 原理扩展了 squid 原有的 COSS 存储引擎而已。所以我们先回忆一下 flashcache 的原理:</p>
<p>flashcache 是利用了 Linux 的 device-mapper 机制来虚拟逻辑块设备,在 ssd 和 sata 设备之间,flashcache 设计了三种模式:</p>
<ol>
<li>Writethrough 模式,<strong>数据同时写到 ssd 和 sata 硬盘</strong>,官方文档的说明是:</li>
</ol>
<blockquote>
<p>safest, all writes are cached to ssd but also written to disk<br />
immediately. If your ssd has slower write performance than your disk (likely<br />
for early generation SSDs purchased in 2008-2010), this may limit your system<br />
write performance. All disk reads are cached (tunable).</p>
</blockquote>
<ol>
<li>Writearound 模式,<strong>数据绕过 ssd,直接写到 sata 设备上</strong>,官方文档的说明是:</li>
</ol>
<blockquote>
<p>again, very safe, writes are not written to ssd but directly to<br />
disk. Disk blocks will only be cached after they are read. All disk reads<br />
are cached (tunable).</p>
</blockquote>
<ol>
<li>Writeback 模式,<strong>数据一开始只写到 ssd 上,然后根据缓存策略再移到 sata 设备上</strong>,官方文档的说明是:</li>
</ol>
<blockquote>
<p>fastest but less safe. Writes only go to the ssd initially, and<br />
based on various policies are written to disk later. All disk reads are<br />
cached (tunable).</p>
</blockquote>
<p>squid-ssd 方案,学习的是 Writeback 模式,这种模式极大的缓解了普通 sata 设备的读写压力,牺牲了一定的数据安全。但是作为 CDN 缓存软件,本身就不需要保证这点 —— 这应该是源站来保证的。</p>
<p>相反,阅读了 ats 的文档说明后,发现 ats 的 interim 方案学习的是 Writearound 模式,而且默认的 tunable 那点还设的比较高, sata 设备上一个缓存对象要累积 2 次读取请求(最低可以修改到1,不能到0)后,才会缓存到 ssd 设备里去。</p>
<p>这一点从另一个细节上也可以反映出来:ats 的监控数据中,<code class="highlighter-rouge">proxy.process.cache.bytes_total</code> 是只计算了 <code class="highlighter-rouge">storage.config</code> 里写的那些 sata 设备容量的,不包括 interim 在的 ssd 设备容量。</p>
【翻译】Kibana 发生什么事了?
2014-02-08T00:00:00+08:00
logstash
kibana
http://chenlinux.com/2014/02/08/whats-cooking-kibana-20140127
<p>注:本文是 Elasticsearch 官方博客 2014 年 1 月 27 日《what’s cooking in kibana》的翻译,原文地址见:<a href="http://www.elasticsearch.org/blog/whats-cooking-kibana/">http://www.elasticsearch.org/blog/whats-cooking-kibana/</a></p>
<hr />
<p>Elasticsearch 1.0 即将发布, Kibana 团队也准备发布自己的新版。除了一些常见的 bug 修复和小调整,下一个版本中还有一些超棒的特性:</p>
<h1 id="section">面板组</h1>
<p>面板现在可以组织成组的形式,组内可以容纳你乐意加入的任意多的面板。每行的删减都很干净,隐藏面板也不会消耗任何资源。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/01/rows_as_groups.png" alt="" /></p>
<h1 id="section-1">图表标记</h1>
<p>变更部署,用户登录以及其他危险性事件导致的流量、内存消耗或者平均负载的变动,图表标记让你可以输入自定义的查询来将这些重要事件标记到时间轴图表上。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/01/chart_markers.png" alt="" /></p>
<h1 id="section-2">即时过滤器</h1>
<p>创建你自己的请求过滤器然后保存下来以备后用。过滤器将和仪表盘一起保存,而且可以在对比你定义的数据子集的时候菜单式展开或收缩。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/01/adhoc_filters.png" alt="" /></p>
<h1 id="top-n-">top-n 查询</h1>
<p>单击某个查询旁边的带色的点,就可以设置这个查询的颜色。新版的top-N 查询会找出一个字段 最流行的结果,然后用他们来完成新的查询。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/01/top_n_queries.png" alt="" /></p>
<h1 id="stats-">stats 面板</h1>
<p>Stats 面板最后都将把搜索归总成一个单独的有意义的数值。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/01/stats_panel.png" alt="" /></p>
<h1 id="termsstats-">terms_stats 模式</h1>
<p>按国家统计流量?每个用户的收入?每页的内存使用?terms面板的terms_stat模式正是你想要的。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2014/01/Screen-Shot-2014-01-27-at-9.14.42-AM.png" alt="" /></p>
Mojo::IOLoop::Delay 模块测试代码解释
2014-01-22T00:00:00+08:00
perl
http://chenlinux.com/2014/01/22/explain-mojo-ioloop-delay-testing
<p>昨天有人在群里问起<a href="https://metacpan.org/source/SRI/Mojolicious-4.68/t/mojo/delay.t">Mojolicious/t/mojo/delay.t</a> 中一段代码的执行原理。代码如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nn">Mojo::</span><span class="nv">Base</span> <span class="o">-</span><span class="nv">strict</span><span class="p">;</span>
<span class="k">BEGIN</span> <span class="p">{</span>
<span class="nv">$ENV</span><span class="p">{</span><span class="nv">MOJO_NO_IPV6</span><span class="p">}</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="nv">$ENV</span><span class="p">{</span><span class="nv">MOJO_REACTOR</span><span class="p">}</span> <span class="o">=</span> <span class="s">'Mojo::Reactor::Poll'</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">use</span> <span class="nn">Test::</span><span class="nv">More</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Mojo::</span><span class="nv">IOLoop</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Mojo::IOLoop::</span><span class="nv">Delay</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$delay</span> <span class="o">=</span> <span class="nn">Mojo::IOLoop::</span><span class="nv">Delay</span><span class="o">-></span><span class="k">new</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$finished</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$result</span> <span class="o">=</span> <span class="nb">undef</span><span class="p">;</span>
<span class="nv">$delay</span><span class="o">-></span><span class="nv">on</span><span class="p">(</span><span class="nv">finish</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="nv">$finished</span><span class="o">++</span> <span class="p">});</span>
<span class="nv">$delay</span><span class="o">-></span><span class="nv">steps</span><span class="p">(</span>
<span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$delay</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$end</span> <span class="o">=</span> <span class="nv">$delay</span><span class="o">-></span><span class="nv">begin</span><span class="p">;</span>
<span class="nv">$delay</span><span class="o">-></span><span class="nv">begin</span><span class="o">-></span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">);</span>
<span class="nn">Mojo::</span><span class="nv">IOLoop</span><span class="o">-></span><span class="nv">timer</span><span class="p">(</span><span class="mi">0</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="nv">$end</span><span class="o">-></span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="p">});</span>
<span class="p">},</span>
<span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$delay</span><span class="p">,</span> <span class="nv">@numbers</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$end</span> <span class="o">=</span> <span class="nv">$delay</span><span class="o">-></span><span class="nv">begin</span><span class="p">;</span>
<span class="nn">Mojo::</span><span class="nv">IOLoop</span><span class="o">-></span><span class="nv">timer</span><span class="p">(</span><span class="mi">0</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="nv">$end</span><span class="o">-></span><span class="p">(</span><span class="nb">undef</span><span class="p">,</span> <span class="nv">@numbers</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span> <span class="p">});</span>
<span class="p">},</span>
<span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$delay</span><span class="p">,</span> <span class="nv">@numbers</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="nv">$result</span> <span class="o">=</span> <span class="o">\</span><span class="nv">@numbers</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">);</span>
<span class="nv">is_deeply</span> <span class="p">[</span><span class="nv">$delay</span><span class="o">-></span><span class="nb">wait</span><span class="p">],</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">4</span><span class="p">],</span> <span class="s">'right return values'</span><span class="p">;</span>
<span class="nv">is</span> <span class="nv">$finished</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="s">'finish event has been emitted once'</span><span class="p">;</span>
<span class="nv">is_deeply</span> <span class="nv">$result</span><span class="p">,</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">4</span><span class="p">],</span> <span class="s">'right results'</span><span class="p">;</span>
<span class="nv">done_testing</span><span class="p">();</span>
</code></pre>
</div>
<p>首先介绍一下这个 <code class="highlighter-rouge">Mojo::IOLoop::Delay</code> 模块,这是异步编程中很火很实用的一个概念,一般叫 <code class="highlighter-rouge">Promise</code> / <code class="highlighter-rouge">Deferred</code> 。你可以按照顺序编程的思路组合那些异步函数,比如在这个例子里主要就体现了 <code class="highlighter-rouge">steps</code> 方法和 <code class="highlighter-rouge">finish</code> 事件。</p>
<p><code class="highlighter-rouge">steps</code> 方法中可以传递任意多个异步函数。第一个函数立刻执行,然后等 <code class="highlighter-rouge">$delay</code> 信号量(由 <code class="highlighter-rouge">begin</code> 方法控制)释放(即重新等于0)后逐次执行后面的函数,直到碰到一个不调用 <code class="highlighter-rouge">begin</code> 控制信号量的函数,或者触发 <code class="highlighter-rouge">error</code> 或者 <code class="highlighter-rouge">finish</code> 事件。</p>
<p><code class="highlighter-rouge">begin</code> 方法返回的回调函数 <code class="highlighter-rouge">$end->()</code> 用来减信号量。如果传递了参数给这个回调函数,那么第一个参数会被忽略,剩下的参数会 <code class="highlighter-rouge">push</code> 进下一个顺序或者事件触发函数的参数列表里,同时推送到 <code class="highlighter-rouge">wait</code> 方法。</p>
<p>所以上面这段测试的数据执行结果是这样的:</p>
<ol>
<li><code class="highlighter-rouge">$delay->wait</code> 开始整个 <code class="highlighter-rouge">ioloop</code>, <code class="highlighter-rouge">steps</code> 方法首先执行 sub1 ,首先通过 <code class="highlighter-rouge">$delay->begin()</code>给信号量加1;</li>
<li>随即触发 <code class="highlighter-rouge">timer</code> 事件,<code class="highlighter-rouge">$end->(1, 2, 3)</code> 将 <code class="highlighter-rouge">(2, 3)</code> 推入下一个函数 sub2 的 <code class="highlighter-rouge">@_</code> 里,同时把信号量减1;</li>
<li>信号量变成0,继续执行,这一行 <code class="highlighter-rouge">$delay->begin()->(3, 2, 1)</code>,将 <code class="highlighter-rouge">(2, 1)</code> 推入下一个函数 sub2 的 <code class="highlighter-rouge">@_</code> 里,注意这里信号量实际也加减过一次,只是这里的回调函数直接匿名调用了;</li>
<li>sub1 执行完成,信号量为0,那么开始下一个sub2,sub2 传入的参数列表其实是 <code class="highlighter-rouge">($delay, (2, 3), (2, 1))</code>,也就是说这时候的 <code class="highlighter-rouge">@numbers</code> 是 <code class="highlighter-rouge">(2, 3, 2, 1)</code>;</li>
<li>sub2 执行流程类似 sub1 ,信号量加1,触发 <code class="highlighter-rouge">timer</code> 事件,然后 <code class="highlighter-rouge">$end->(undef, @numbers, 4)</code> 把 <code class="highlighter-rouge">((2, 3, 2, 1), 4)</code> 推入下一个函数 sub3 的 <code class="highlighter-rouge">@_</code> 里,同时信号量减1;</li>
<li>sub2 执行完成,信号量为0,那么开始下一个sub3,sub3 传入的参数列表就是 <code class="highlighter-rouge">($delay, (2, 3, 2, 1, 4))</code>,也就是说这时候的 <code class="highlighter-rouge">@numbers</code> 是 <code class="highlighter-rouge">(2, 3, 2, 1, 4)</code>;</li>
<li>sub3 将 <code class="highlighter-rouge">@numbers</code> 的引用赋值给 <code class="highlighter-rouge">$result</code>,因为 sub3 里没有对信号量的操作,而且也是最后一个了,<code class="highlighter-rouge">steps</code> 完成,触发 <code class="highlighter-rouge">finish</code> 事件;</li>
<li>注册的 <code class="highlighter-rouge">finish</code> 事件回调函数把 <code class="highlighter-rouge">$finish</code> 变量加1;</li>
<li><code class="highlighter-rouge">$delay->wait</code> 这时候也收集完毕前面每个 <code class="highlighter-rouge">$end->()</code> 的参数列表,和每步 <code class="highlighter-rouge">@numbers</code> 是同步的,同时因为 <code class="highlighter-rouge">finish</code> 事件被触发,就此停止 <code class="highlighter-rouge">ioloop</code>,程序完成,返回整个列表。</li>
</ol>
<p>如上。</p>
【翻译】Kibana3 里程碑 4
2014-01-15T00:00:00+08:00
logstash
elasticsearch
kibana
http://chenlinux.com/2014/01/15/kibana3-milestone4-20131105
<p>本文来自Elasticsearch官方博客,2013年11月5日的文章<a href="http://www.elasticsearch.org/blog/kibana-3-milestone-4/">Kibana 3: mileston 4</a>,作为kibana3 Milestone 4重要的使用说明,翻译如下:</p>
<p>Kibana 3: Milestone 4 已经发布,带来了一系列性能、易用性和可视化上的提升。让我们来看看这些重大改变。如果你还在Milestone 3上,先看看之前<a href="http://chenlinux.com/2014/01/14/this-week-in-kibana-20130919">这篇博客</a>里的新特性介绍。</p>
<h1 id="section">一个全新的界面</h1>
<p>Kibana 面板改造成了一个标签更突出,按键和链接更易用,风格全新的样子。改造结果提高了可用度,因为有了更高效的空间利用设计,来支持更大的数据密度和更一致的UI。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/11/Screen-Shot-2013-10-31-at-3.45.06-PM.png" alt="Kibana的新界面" /></p>
<h1 id="section-1">一致性查询和过滤布局</h1>
<p>为了改善UI,查询和过滤面板现在有自己的可折叠、下拉的区域,具体位置在导航栏的下方。以后不再需要你自己摆放这些基本面板的布局了,它们默认会包含在每一个仪表盘里。和很多Kibana的特性一样,你也可以在仪表盘配置对话框里禁用这个一致性布局。</p>
<h1 id="section-2">100%全新的时间范围选择器</h1>
<p>如果你熟悉Kibana这两年来的历史,你可能知道曾经存在过好几个时间选择器方案。新的时间选择器经过了完全的重写,不仅占用空间比原来的小,也更容易使用。把这个重要组件移出主仪表盘后,Kibana 现在有更多空间专注于重要数据和图表。还有,新的过滤格式实现了Elasticsearch的<a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-date-format.html#date-math">时间运算</a>,所以不用每次重新选择一个时间范围来移动你的时间窗口了,每个搜索都能自动更新这个窗口。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/11/Screen-Shot-2013-10-31-at-3.44.17-PM.png" alt="全新的时间选择器" /></p>
<h1 id="section-3">可过滤的字段列表</h1>
<p>利用表格的”即输即过滤”特性,可以简单而快速的找到字段。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/11/Screen-Shot-2013-10-31-at-3.46.52-PM.png" alt="可过滤的字段列表" /></p>
<h1 id="ad-hoc-facets">即时(ad-hoc) facets</h1>
<p>然后,当你找到了这些字段,就可以利用即时 facets 快速分析他们。只需要点击一个字段然后选择可视化即可查看到前10个匹配该字段的term。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/11/Screen-Shot-2013-10-31-at-3.45.42-PM.png" alt="" /></p>
<p>研究起来也更加简单了</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/11/Screen-Shot-2013-10-31-at-3.45.57-PM.png" alt="" /></p>
<p>不需要添加面板,饼图可以直接悬浮出现!</p>
<h1 id="url">动态的仪表盘和url参数</h1>
<p>Kibana 3: Milestone 4现在可以通过URL参数获取输入!这个备受期待的特性体现为两个方式:模板化的仪表盘和脚本化的仪表盘。Kibana 3: Milestone 4附带两个可以和Logstash完美配合的示例,在此基础上你可以构建自己的仪表盘。模板化仪表盘的创建非常简单,导出当前仪表盘结构成文件,编辑文件然后保存添加进你的 app/dashboards 目录既可以了。比如,从 <a href="https://github.com/elasticsearch/kibana/blob/v3.0.0milestone4/src/app/dashboards/logstash.json">logstash.json</a> 里摘录下面一段:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="w"> </span><span class="s2">"0"</span><span class="err">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"query"</span><span class="p">:</span><span class="w"> </span><span class="s2">"{<span>{</span>ARGS.query || '*'}}"</span><span class="p">,</span><span class="w">
</span><span class="nt">"alias"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span><span class="w">
</span><span class="nt">"color"</span><span class="p">:</span><span class="w"> </span><span class="s2">"#7EB26D"</span><span class="p">,</span><span class="w">
</span><span class="nt">"id"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w">
</span><span class="nt">"pin"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
</div>
<p>模板化仪表盘用”handlebar 语法”添加动态区段到基于JSON的仪表盘结构里。比如这里我们就用一个表达式替换掉了查询键的内容:<em>使用URL里的请求参数,如果不存在,使用’*‘。</em> 现在我们可以用下面这条URL访问这个仪表盘了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>http://kibana.example.com/index.html#/dashboard/file/logstash.json?query=extension:zip
</code></pre>
</div>
<h1 id="section-4">更灵活的脚本化仪表盘</h1>
<p>脚本化仪表盘在处理URL参数的时候更加强大,它能运用上Javascript的全部威力构建一个完整的仪表盘对象。同样用 app/dashboards 里的 <a href="https://github.com/elasticsearch/kibana/blob/v3.0.0milestone4/src/app/dashboards/logstash.js">logstash.js</a> 举例。因为脚本化仪表盘完全就是javascript,我们可以执行复杂的操作,比如切割URL参数。如下URL中,我们搜索_最近2天内的<code class="highlighter-rouge">HTML</code>, <code class="highlighter-rouge">CSS</code> 或者 <code class="highlighter-rouge">PHP</code>,然后在表格里显示 <code class="highlighter-rouge">request</code>, <code class="highlighter-rouge">response</code> 和 <code class="highlighter-rouge">user agent</code>。_注意URL本身路径从 <strong>file__变成了__script</strong>:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>http://localhost:8000/index.html#/dashboard/script/logstash.js?query=html,css,php&from=2d&fields=request,response,agent
</code></pre>
</div>
<h1 id="section-5">立刻下载</h1>
<p>Milestone 4对作者和使用者都是一个飞跃。它功能更强大,当然使用也更简单。Kibana 继续集成在 Logstash 里,最新发布的 <a href="http://logstash.net/docs/1.2.2/">Logstash 1.2.2</a> 中就带有。Kibana现在也可以直接用elasticsearch.org官网下载,地址见:<a href="http://www.elasticsearch.org/overview/kibana/installation/">http://www.elasticsearch.org/overview/kibana/installation/</a>。</p>
【翻译】2013 年 9 月的 kibana 周报
2014-01-14T00:00:00+08:00
logstash
elasticsearch
kibana
http://chenlinux.com/2014/01/14/this-week-in-kibana-20130919
<p>本文来自Elasticsearch官方博客,2013年9月19日的文章<a href="http://www.elasticsearch.org/blog/this-week-in-kibana/">this week in kibana</a>,作为kibana3 Milestone 3重要的使用说明,翻译如下:</p>
<h1 id="section">直方图零填充</h1>
<p>直方图面板经过了一番改造,实现了正确的零填充。也就是说,当一个间隔内查询收到0个结果的时候,就显示为0,而不是绘制一条斜线连接到下一个点。零填充也意味着堆叠式直方图从顶端到底部的次序将保持不变。</p>
<p>此外,堆叠提示栏现在允许你在累积和个人模式之间自由选择。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/09/Screen-Shot-2013-09-18-at-3.13.27-PM.png" alt="" /></p>
<h1 id="section-1">数组字段的微分析</h1>
<p>数组字段现在可以在微分析面板上单独或者分组处理。比如,如果我有一个tags数组,我即可以看到前10个最常见的tags,也可以看到前10个最常见的tags组合。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/09/Screen-Shot-2013-09-18-at-3.16.07-PM.png" alt="" /><br />
<img src="http://www.elasticsearch.org/content/uploads/2013/09/Screen-Shot-2013-09-18-at-3.16.21-PM.png" alt="" /></p>
<h1 id="source-"><code class="highlighter-rouge">_source</code> 作为默认的表字段</h1>
<p>如果你没有给你的表选择任何字段,Kibana现在默认会给你显示 <code class="highlighter-rouge">_source</code> 里的 json 数据,直到你选择了具体的字段。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/09/Screen-Shot-2013-09-18-at-3.14.00-PM.png" alt="" /></p>
<h1 id="section-2">可配置的字段截取</h1>
<p>注意到下面截图中 <code class="highlighter-rouge">_source</code> 字段末尾的”…“了吗?表格字段能被一个可以配置的”因子”截断。所谓因子就是,表格的列数除以它,得到一个字段的最大长度,然后各字段会被很好的截断成刚好符合这个长度。比如,如果我的截断因子是300,而表格有3列,那么每个字段会被截断成最大100个字符,然后后面跟上’…‘。当然,字段的完整内容还是可以在细节扩展视图里看到的。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/09/Screen-Shot-2013-09-18-at-3.17.19-PM.png" alt="" /></p>
<h1 id="section-3">关于细节视图</h1>
<p>你可能已经知道单击表格某行后可以看到包含这个事件的字段的表格。现在你可以选择你希望如何观察这个事件的细节了,包括有语法高亮的JSON以及原始的未高亮的JSON。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/09/Screen-Shot-2013-09-18-at-3.17.47-PM.png" alt="" /></p>
<h1 id="section-4">更轻,更快,更小,更好</h1>
<p>Kibana有了一个全新的构建系统!新的系统允许我们构建一个优化的,小巧的,漂亮的新Kibana。当你升级的时候它还可以自动清除原来的缓存,定期构建的Kibana发布在 <a href="http://download.elasticsearch.org/kibana/kibana/kibana-latest.zip">http://download.elasticsearch.org/kibana/kibana/kibana-latest.zip</a> ,zip包可以直接解压到你的web服务器里。</p>
<p>如果愿意,你也可以从 <a href="https://github.com/elasticsearch/kibana">Github repository</a> 开始运行。不用复制整个项目,只需要上传 src/ 目录到服务器就可以了。不过我们强烈建议使用构建好的版本,因为这样性能好很多。</p>
【翻译】kibana发生什么变化了?
2014-01-14T00:00:00+08:00
logstash
elasticsearch
kibana
http://chenlinux.com/2014/01/14/kibana-what-happened
<p>本文来自Elasticsearch官方博客,2013年8月21日的文章<a href="http://www.elasticsearch.org/blog/kibana-whats-cooking/">kibana: what’s cooking</a>,作为kibana3重要的使用说明,翻译如下:</p>
<p>还没有升级Kibana么?那你可错过了一个好技术!Kibana 发生了翻天覆地的变化,新面板只是这个故事中的一部分。整个系统都被重构,给表盘提供统一的颜色和图例方案选择。接口也经过了标准化,很多函数都修改成提供更简单,快速和功能更强大的方式。让我们进一步看看现在的样子。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/08/BQIielHCAAAs2So.png" alt="" /></p>
<p>Terms 面板;全局色彩;别名和查询;过滤器。</p>
<h1 id="section">新的查询输入</h1>
<p>新的查询面板替代了原来的“字符串查询”面板作为你输入查询的方式。每个面板都有自己独立的请求输入。你也还可以为特殊的面板定制请求,不过你要先在这里输入他们,包括可以有别名和颜色设置,然后再在面板编辑器里选取。在没有被激活修改的时候, 查询也可以被固定在一个可折叠的区域。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/08/Screen-Shot-2013-08-20-at-11.48.43-AM.png" alt="" /></p>
<h1 id="section-1">分配查询到具体面板</h1>
<p>分配查询到具体面板非常非常简单。面板编辑器里就可以直接打开或关闭查询,哪怕这个查询已经更新或者过滤掉,它的别名是保持全局一致性的。你还会注意到配置窗口被分割成了选项卡形式,已提供更清晰的配置界面。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/08/Screen-Shot-2013-08-20-at-1.34.08-PM.png" alt="" /></p>
<h1 id="section-2">自定义颜色和别名</h1>
<p>当你给一个查询分配某个颜色的时候,它会立刻反映到所有的面板上。通常用于做图例值的别名也一样。这样,我们可以很简单的通过在一个逻辑组里分配颜色变化,调节整个仪表盘和数据的意义。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/08/Screen-Shot-2013-07-11-at-5.00.28-PM.png" alt="" /></p>
<h1 id="terms">你好,terms!</h1>
<p>引入了一个新的terms面板,可以使用3种不同的格式展示顶层字段数据:饼图、柱状图和表格。而且都可以点击进入新的过滤器面板。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/08/Screen-Shot-2013-08-20-at-1.47.56-PM.png" alt="" /></p>
<h1 id="section-3">过滤器面板?</h1>
<p>刚刚提到过滤器面板,对吧?没错,过滤器!过滤器允许你深入分解数据集而不用你去修改查询本身。然后,过滤器也可以被删除、隐藏和编辑。过滤器有三种模式:</p>
<ul>
<li><strong>must</strong>: 记录必须匹配这个过滤器;</li>
<li><strong>mustNot</strong>: 记录必须不能匹配这个过滤器;</li>
<li><strong>either</strong>: 记录必须匹配这些过滤器中的一个。</li>
</ul>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/08/Screen-Shot-2013-08-20-at-1.55.54-PM.png" alt="" /></p>
<h1 id="section-4">字段列表和微面板</h1>
<p>字段面板集成在表格面板里。字段列表现在会通过访问Elasticsearch的<code class="highlighter-rouge">/_mapping</code>API来自动填充。注意你可能需要更新自己的代理服务器配置来适应这个变更。为了节约空间,这个字段列表现在也是可折叠的,而新的图形也添加到了微面板。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/08/Screen-Shot-2013-08-20-at-7.56.02-AM.png" alt="" /></p>
<h1 id="section-5">嗨,那配色方案呢?!</h1>
<p>对,你在我解释之前已经发现这个变化了!Kibana现在允许你在黑白两个配色方案之间切换以刚好的匹配你自己的环境和偏好。</p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/08/BQjv-50CcAAyazu.png" alt="" /></p>
<p>汇报完毕!当然kibana一直在更新,注意继续关注这里,给我们的<a href="https://github.com/elasticsearch/kibana/">github项目</a>加星,然后上推特fo <a href="https://twitter.com/rashidkpc/">@rashidkpc</a> 和 <a href="https://twitter.com/elasticsearch/">@elasticsearch</a>。</p>
私有 docker 仓库部署测试
2014-01-08T00:00:00+08:00
cloud
docker
python
http://chenlinux.com/2014/01/08/run-docker-registry
<p>docker 的官方仓库 CDN 的ip 总是被 GFW 认证。为了更好的使用 docker ,有必要在自己内部搭建一个私有仓库。方法很简单:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">git</span> <span class="n">clone</span> <span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">github</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">dotcloud</span><span class="o">/</span><span class="n">docker</span><span class="o">-</span><span class="n">registry</span><span class="o">.</span><span class="n">git</span>
<span class="n">cd</span> <span class="n">docker</span><span class="o">-</span><span class="n">registry</span>
<span class="c"># 安装依赖</span>
<span class="n">yum</span> <span class="n">install</span> <span class="n">python</span><span class="o">-</span><span class="n">devel</span> <span class="n">libevent</span><span class="o">-</span><span class="n">devel</span> <span class="n">python</span><span class="o">-</span><span class="n">pip</span> <span class="n">openssl</span><span class="o">-</span><span class="n">devel</span> <span class="n">xz</span><span class="o">-</span><span class="n">devel</span> <span class="o">--</span><span class="n">enablerepo</span><span class="o">=</span><span class="n">epel</span>
<span class="n">python</span><span class="o">-</span><span class="n">pip</span> <span class="n">install</span> <span class="o">-</span><span class="n">r</span> <span class="n">requirements</span><span class="o">.</span><span class="n">txt</span>
<span class="c"># 默认读取config/config.yml里的dev配置</span>
<span class="n">WORKER_SECRET_KEY</span><span class="o">=</span><span class="s">"${WORKER_SECRET_KEY:-$(< /dev/urandom tr -dc A-Za-z0-9 | head -c 32)}"</span>
<span class="n">cat</span> <span class="o">></span> <span class="n">config</span><span class="o">/</span><span class="n">config</span><span class="o">.</span><span class="n">yml</span><span class="o"><</span><span class="n">EOF</span>
<span class="n">dev</span><span class="p">:</span>
<span class="n">storage</span><span class="p">:</span> <span class="n">local</span>
<span class="n">storage_path</span><span class="p">:</span> <span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">registry</span>
<span class="n">secret_key</span><span class="p">:</span> <span class="err">$</span><span class="p">{</span><span class="n">WORKER_SECRET_KEY</span><span class="p">}</span>
<span class="n">EOF</span>
<span class="c"># 默认的镜像存储位置,可以在 config.yml 里更改 storage_path</span>
<span class="n">mkdir</span> <span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">registry</span>
<span class="c"># 默认监听5000端口,前台运行,可以加入daemontools、supervisor、ubic之类的来负责</span>
<span class="n">sh</span> <span class="n">run</span><span class="o">.</span><span class="n">sh</span>
</code></pre>
</div>
<p>这就完成了。如果想用 nginx 作代理和加速镜像下载性能的,代码里也提供了 nginx.conf 可用。不过注意要求 nginx 版本在 1.3.9 以上,同时编译的时候还要加上 chunkin 模块。否则镜像上传的时候会出错。</p>
<p>然后就是客户端如何指定镜像推送到私有仓库里:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># 在私有仓库注册用户</span>
docker login 127.0.0.1:5000
<span class="c"># 给要提交的镜像打标签</span>
docker tag <IMAGE ID> 127.0.0.1:5000/tagname
<span class="c"># 推送到私有仓库</span>
docker push 127.0.0.1:5000/tagname
</code></pre>
</div>
<p>注意这里推送的时候使用的是REPOSITORY,也就是说不能是 <code class="highlighter-rouge">127.0.0.1:5000/ubuntu:12.04</code> 这样的格式。</p>
<p>现在就可以在其他地方用了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>docker pull 192.168.0.2:5000/tagname
</code></pre>
</div>
利用 staticperl 和 upx 生成 单个可执行 perl
2014-01-06T00:00:00+08:00
http://chenlinux.com/2014/01/06/staticperl-and-upx
<p>Perl 程序打包的问题由来已久。</p>
<p>最早是 perlcc,但是从5.10版本以后,B::CC 等一系列模块跟不上开发脚本导致 perlcc 也无法使用。</p>
<p>然后是PAR::Packer,唐凤大神的作品。</p>
<p>今天介绍另一个模块,App::Staticperl,同样是大神级作品,作者是Marc Lehmann。他的 AnyEvent、Coro、EV 无不大名鼎鼎。而staticperl,就是他开发出来用以方便自己部署程序的。</p>
<p>staticperl 官网上有一句很霸气的描述:“perl, libc, 100 modules, all in one standalone 500kb file”。</p>
<p>不过经我测试,按照官网上的步骤是做不出来这么小的单文件的!幸运的是我在 Perlmonks 上的<a href="http://www.perlmonks.org/?node_id=1065912">发问</a>很快收到了答案,这个还要用上另一个工具:upx。</p>
<p>测试过程如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># cpanm App::Staticperl</span>
<span class="c"># staticperl install</span>
<span class="c"># staticperl instcpan AnyEvent AnyEvent::HTTP</span>
<span class="c"># staticperl mkperl -MAnyEvent -MAnyEvent::HTTP</span>
<span class="c"># staticperl mkapp myapp --boot myapp.pl -MAnyEvent -MAnyEvent::HTTP</span>
</code></pre>
</div>
<p>而如果是官网说的 <a href="http://staticperl.schmorp.de/smallperl.html">smallperl</a>,则是采用 <code class="highlighter-rouge">mkbundle</code> 的方法。</p>
<p>除了使用单独的<a href="http://staticperl.schmorp.de/smallperl.bundle">配置文件</a>存放太长的参数,其他和 <code class="highlighter-rouge">mkapp</code> / <code class="highlighter-rouge">mkperl</code> 一致。</p>
<p>不过运行结果是:生成的单个文件有3.5MB大小。</p>
<p>然后使用 upx:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># apt-get install upx</span>
<span class="c"># upx --best smallperl.bin</span>
</code></pre>
</div>
<p>就得到压缩后的超小型perl了。这个perl内含了AE、Socket、common::sense、List::Util 等一系列常用模块可以直接使用。不过大小依然有 1.7MB 。看来是 Perl5.14 本身大小也变大了。</p>
<p><strong>补充</strong></p>
<p>按照评论里的建议,改用 <code class="highlighter-rouge">--lzma</code> 选项再压缩一次:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># upx -d smallperl.bin</span>
<span class="c"># upx --lzma smallperl.bin</span>
</code></pre>
</div>
<p>结果到 1.4MB 大小。</p>
通过网页运行 Perl 代码的安全实现
2014-01-05T00:00:00+08:00
perl
perl
docker
javascript
http://chenlinux.com/2014/01/05/run-perl-code-from-webpage
<p>这几天折腾<a href="http://www.perl-china.com">Perl中国用户组网站</a>,觉得类似 Ruby 的 tryruby,Scala 的 scala-tour 这样的新手入门教程非常好玩。于是准备自己也尝试一下。</p>
<p>理论上,通过 Ajax 传递代码到服务器上,直接 <code class="highlighter-rouge">eval {}</code> 即可。不过这样会导致一个安全问题。如何防止用户执行错误代码导致严重后果呢?</p>
<p>我想到了最近一直在跟踪看的 Docker 容器。如果我们把代码放在 Docker 里运行,不就不怕了么。</p>
<p>首先要构建一个可以运行大多数示例代码的 Docker 镜像。</p>
<h3 id="section">首先打开一个终端运行初始镜像:</h3>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># docker run -i -t ubuntu /bin/sh</span>
<span class="c"># apt-get install -y wget gcc make</span>
<span class="c"># useradd tour</span>
<span class="c"># echo 'tour hard nproc 8' >> /etc/security/limits.conf</span>
<span class="c"># wget http://cpanmin.us -O bin/cpanm</span>
<span class="c"># cpanm List::AllUtils Moo Path::Tiny DBD::SQLite AnyEvent::HTTP DateTime</span>
</code></pre>
</div>
<h3 id="section-1">然后打开另一个终端保存前一个终端的变更:</h3>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># docker ps</span>
CONTAINER ID ...
<span class="c"># docker commit <ID> perl-tour</span>
</code></pre>
</div>
<p>注意一定要在之前 <code class="highlighter-rouge">cpanm</code> 已经成功执行完毕后保存,但是前面登录进 docker 的会话千万不要退出,否则后面的 <code class="highlighter-rouge">docker ps</code> 就查看不到 id 了。退出时这些临时变更都毁掉了。</p>
<p><strong>2014 年 1 月 7 日补充</strong></p>
<p>被莫莫用死循环 <code class="highlighter-rouge">fork()</code> 轰炸了一回,发现 docker 容器的一个问题,容器技术本身没有对用户最大进程数的限制。因为其实际运行的是 <code class="highlighter-rouge">docker -d</code> 服务进程的子进程。</p>
<p>直接在镜像里编辑 <code class="highlighter-rouge">/etc/security/limits.conf</code> 实测没有作用。而主机上限定普通用户的 nproc 也没用(因为普通用户运行不了 docker )。</p>
<p>最后想到的办法,是启动 <code class="highlighter-rouge">docker -d</code> 的时候,先 <code class="highlighter-rouge">ulimit -HSu 16</code>,这样这个 docker 下一共也跑不了多少 fork 了。</p>
<p>顺带提一句,查阅系统日志可以发现,在 fork 的时候,其实触发了主机的 OOM-killer,但是这个机制在死循环这个变态攻击下挽救不了主机……</p>
<p><strong>END</strong></p>
<p>现在我们已经有了一个安装好很多常用 CPAN 模块的镜像了。可以取构建网站了。</p>
<p>网站里添加下面一段:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nn">Dancer::Plugin::</span><span class="nv">Ajax</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">File::</span><span class="nv">Temp</span> <span class="sx">qw(tempfile)</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">IPC::</span><span class="nv">Run</span> <span class="sx">qw(start harness timeout)</span><span class="p">;</span>
<span class="nv">ajax</span> <span class="s">'/run'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$code</span> <span class="o">=</span> <span class="nv">param</span><span class="p">(</span><span class="s">'code'</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">@cmd</span> <span class="o">=</span> <span class="sx">qw(docker run -m 128m -u tour -v /tmp/:/tmp:ro perl-tour perl)</span><span class="p">;</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$fh</span><span class="p">,</span> <span class="nv">$temp</span><span class="p">)</span> <span class="o">=</span> <span class="nv">tempfile</span><span class="p">();</span>
<span class="nb">binmode</span><span class="p">(</span><span class="nv">$fh</span><span class="p">,</span> <span class="s">':utf8'</span><span class="p">);</span>
<span class="k">print</span> <span class="nv">$fh</span> <span class="nv">$code</span><span class="p">;</span>
<span class="nb">push</span> <span class="nv">@cmd</span><span class="p">,</span> <span class="nv">$temp</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$h</span><span class="p">;</span>
<span class="nb">eval</span> <span class="p">{</span>
<span class="nv">$h</span> <span class="o">=</span> <span class="nv">harness</span> <span class="o">\</span><span class="nv">@cmd</span><span class="p">,</span> <span class="o">\</span><span class="nv">$in</span><span class="p">,</span> <span class="o">\</span><span class="nv">$out</span><span class="p">,</span> <span class="o">\</span><span class="nv">$err</span><span class="p">,</span> <span class="nv">timeout</span><span class="p">(</span><span class="mi">5</span><span class="p">);</span>
<span class="nv">start</span> <span class="nv">$h</span><span class="p">;</span>
<span class="nv">$h</span><span class="o">-></span><span class="nv">finish</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">if</span><span class="p">(</span><span class="vg">$@</span><span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$x</span> <span class="o">=</span> <span class="vg">$@</span><span class="p">;</span>
<span class="nv">$h</span><span class="o">-></span><span class="nv">kill_kill</span><span class="p">;</span>
<span class="k">return</span> <span class="nv">$x</span><span class="p">;</span>
<span class="p">};</span>
<span class="nb">unlink</span> <span class="nv">$temp</span><span class="p">;</span>
<span class="k">return</span> <span class="nv">to_json</span><span class="p">({</span>
<span class="nv">Errors</span> <span class="o">=></span> <span class="p">[</span> <span class="nb">split</span><span class="p">(</span><span class="sr">/\n/</span><span class="p">,</span> <span class="nv">$err</span><span class="p">)</span> <span class="p">],</span>
<span class="nv">Events</span> <span class="o">=></span> <span class="p">[</span> <span class="nb">split</span><span class="p">(</span><span class="sr">/\n/</span><span class="p">,</span> <span class="nv">$out</span><span class="p">)</span> <span class="p">],</span>
<span class="p">});</span>
<span class="p">};</span>
</code></pre>
</div>
<p>页面上通过 Ajax 请求交互:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nx">$</span><span class="p">.</span><span class="nx">ajax</span><span class="p">(</span><span class="s2">"/run?code="</span> <span class="o">+</span> <span class="nb">encodeURIComponent</span><span class="p">(</span><span class="nx">codeStr</span><span class="p">),</span> <span class="p">{</span>
<span class="na">type</span><span class="p">:</span> <span class="s2">"GET"</span><span class="p">,</span>
<span class="na">dataType</span><span class="p">:</span> <span class="s2">"json"</span><span class="p">,</span>
<span class="na">success</span><span class="p">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">data</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">data</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">data</span><span class="p">.</span><span class="nx">Errors</span> <span class="o">&&</span> <span class="nx">data</span><span class="p">.</span><span class="nx">Errors</span><span class="p">.</span><span class="nx">length</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">setOutput</span><span class="p">(</span><span class="nx">outputDiv</span><span class="p">,</span> <span class="kc">null</span><span class="p">,</span> <span class="kc">null</span><span class="p">,</span> <span class="nx">data</span><span class="p">.</span><span class="nx">Errors</span><span class="p">);</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="nx">setOutput</span><span class="p">(</span><span class="nx">outputDiv</span><span class="p">,</span> <span class="nx">data</span><span class="p">.</span><span class="nx">Events</span><span class="p">,</span> <span class="nx">data</span><span class="p">.</span><span class="nx">ErrEvents</span><span class="p">,</span> <span class="kc">false</span><span class="p">);</span>
<span class="p">},</span>
<span class="na">error</span><span class="p">:</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">outputDiv</span><span class="p">.</span><span class="nx">addClass</span><span class="p">(</span><span class="s2">"error"</span><span class="p">).</span><span class="nx">text</span><span class="p">(</span>
<span class="s2">"Error communicating with remote server."</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">});</span>
</code></pre>
</div>
<p>静态页面部分严重参考了 Scala 的 Tour 页。趁机学习了 impress.js 制作幻灯片效果、codemirror 实现代码高亮效果。</p>
<p>最终效果见 <a href="http://www.perl-china.com/tour.html">少年 Perl 的魔法世界</a>。欢迎大家莅临指导~</p>
<p>最后,阅读了 Golang Tour 关于 <a href="http://play.golang.org">Go Playground</a> 的原理说明,发现它们是在 Google App Engine 上运行实例,然后走消息队列把代码发送给后台实例运行结果。</p>
<p>当然,Go Playground 不单单是支持 Tour,而且还包括社区各式第三方模块的测试和使用。把角色拆分出来也是正常的。</p>
Future模块和AnyEvent事件驱动的结合
2014-01-05T00:00:00+08:00
perl
anyevent
future
http://chenlinux.com/2014/01/05/future-with-anyevent
<p>上个月的 advent calendar 活动中,有一个新的模块进入我们视野,这就是 IO::Async 模块作者写的 <a href="https://metacpan.org/pod/Future">Future</a> 模块。通过 Future 模块,我们可以做到对异步请求的各种控制,比如:</p>
<ul>
<li><code class="highlighter-rouge">needs_all</code> / <code class="highlighter-rouge">needs_any</code> / <code class="highlighter-rouge">wait_any</code> / <code class="highlighter-rouge">wait_all</code></li>
<li><code class="highlighter-rouge">then</code> / <code class="highlighter-rouge">else</code> / <code class="highlighter-rouge">and_then</code> / <code class="highlighter-rouge">or_else</code> / <code class="highlighter-rouge">followed_by</code></li>
<li><code class="highlighter-rouge">on_ready</code> / <code class="highlighter-rouge">on_done</code> / <code class="highlighter-rouge">on_fail</code> / <code class="highlighter-rouge">on_cancel</code></li>
</ul>
<p>目前来说,IO::Async 是原生支持 Future 了的。但是 AnyEvent 框架才是目前 Perl 社区事件驱动编程的主流选择。还好 Future 源码目录下 <a href="https://metacpan.org/source/PEVANS/Future-0.21/examples">examples/</a> 里有关于 AnyEvent 和 POE 如何跟 Future 一起运行的示例。</p>
<p>示例统一举例的是 timer 事件。而我更看好的是 <a href="https://metacpan.org/pod/Future::Utils">Future::Utils</a> 提供的一些关于循环的函数,比如 <code class="highlighter-rouge">fmap</code> 可以很简单的控制住异步的并发数。稍微试验,得到脚本如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nb">package</span> <span class="nn">Future::</span><span class="nv">AnyEvent</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">base</span> <span class="sx">qw( Future )</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">AnyEvent</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">AnyEvent::</span><span class="nv">HTTP</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">await</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$self</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$cv</span> <span class="o">=</span> <span class="nv">AnyEvent</span><span class="o">-></span><span class="nv">condvar</span><span class="p">;</span>
<span class="nv">$self</span><span class="o">-></span><span class="nv">on_ready</span><span class="p">(</span><span class="k">sub </span><span class="p">{</span> <span class="nv">$cv</span><span class="o">-></span><span class="nb">send</span> <span class="p">});</span>
<span class="nv">$cv</span><span class="o">-></span><span class="nb">recv</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">httpget</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$self</span> <span class="o">=</span> <span class="nb">shift</span><span class="o">-></span><span class="k">new</span><span class="p">;</span>
<span class="nv">http_get</span><span class="p">(</span><span class="nb">shift</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$content</span><span class="p">,</span> <span class="nv">$headers</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="nv">$self</span><span class="o">-></span><span class="nv">done</span><span class="p">(</span><span class="nv">$content</span><span class="p">);</span>
<span class="p">});</span>
<span class="k">return</span> <span class="nv">$self</span><span class="p">;</span>
<span class="p">}</span>
<span class="nb">package</span> <span class="nv">main</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Future::</span><span class="nv">Utils</span> <span class="sx">qw/fmap/</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@urls</span> <span class="o">=</span> <span class="sx">qw(
http://www.sina.com.cn
http://www.baidu.com
http://www.sohu.com
# ...
)</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$f</span> <span class="o">=</span> <span class="nv">fmap</span> <span class="p">{</span>
<span class="nn">Future::</span><span class="nv">AnyEvent</span><span class="o">-></span><span class="nv">httpget</span><span class="p">(</span> <span class="nb">shift</span> <span class="p">);</span>
<span class="p">}</span> <span class="k">foreach</span> <span class="o">=></span> <span class="o">\</span><span class="nv">@urls</span><span class="p">,</span> <span class="nv">concurrent</span> <span class="o">=></span> <span class="mi">5</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@res</span> <span class="o">=</span> <span class="nv">$f</span><span class="o">-></span><span class="nv">get</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">@res</span><span class="p">;</span>
</code></pre>
</div>
<p>看起来稍显复杂。这里其实最关键的就是几个接口函数:</p>
<ul>
<li>await / on_ready</li>
</ul>
<p>Future 对象到实际执行时(即->get调用处),会寻找 <code class="highlighter-rouge">await</code> 方法。所以必须给自己选用的事件驱动实现这个 <code class="highlighter-rouge">await</code> 方法。</p>
<p>ready 状态即一个 Future 执行完成,注意执行完成不意味着执行成功,ready 状态包括 success 和 fail 两种,其实是可以分别定义 <code class="highlighter-rouge">on_success</code> 和 <code class="highlighter-rouge">on_failure</code> 回调的。<br />
<code class="highlighter-rouge">on_ready</code> 回调的作用是:在该 Future 对象达到 ready 状态的时候,执行这步调用。</p>
<p>在本例使用 AnyEvent 的时候,也就是一般来说都会在每步操作结束的 <code class="highlighter-rouge">$cv->send</code> 改到这里来等待调用。</p>
<ul>
<li>done / done_cb</li>
</ul>
<p>那 Future 对象的 ready 状态是怎么来的呢?就是这步了:<code class="highlighter-rouge">$f->done</code> 一旦被调用,就意味着该 Future 对象进入了 ready and success 状态。</p>
<p>同样,如果你要详细控制 Future 对象进入具体的 ready but failure 状态,就使用 <code class="highlighter-rouge">$f->fail</code> 好了。</p>
<p>调用 <code class="highlighter-rouge">->done|fail()</code> 的时候,你可以选择传递具体哪些数据。比如本例中,就只传递了抓取的 <code class="highlighter-rouge">$content</code> 而没有 <code class="highlighter-rouge">$headers</code>。</p>
<p>Future 提供了 <code class="highlighter-rouge">->done_cb</code> 和 <code class="highlighter-rouge">->fail_cb</code> 两个回调函数,默认传递回当前全部数据。本例如果要传回全部,就可以直接写成<code class="highlighter-rouge">http_get shift, $self->done_cb</code>。</p>
<p>好了,就到这里。这个例子虽然比 Future 自带的 <code class="highlighter-rouge">anyevent.pl</code> 示例稍微复杂一点,但是依然很简单。如果能引起大家的兴趣,请直接阅读官方文档。</p>
2013 年度个人总结
2013-12-31T00:00:00+08:00
http://chenlinux.com/2013/12/31/report-of-this-year
<p>又到了一年年底。照例(虽然这个例也就是去年开始的)开始年度总结。</p>
<p>在一年前写总结时,我决然想不到今年会是这样。事实上,当初的计划是往底层深入学习,在 Linux 或者 TCP/IP 方面有所得。但是一年后现在看,今年的工作依然集中在 Puppet 和 监控两方面。所以今年盘点,可能能让自己记忆深刻的,大多“功夫在课外”了。</p>
<p>年初,在编辑鼓励下开始尝试整理过去的知识体系,准备写一本网站运维相关的书籍。感谢这几年坚持记录博客,大几百的 Word 文档最后还比原计划提前写完了。写书的几个月中,我总是开玩笑的说“赚点钱买的起沙发就满足了”,虽然书还没出版,但其实这个过程中本身的收获已然很多。网上很多大神们说写书没意义,或许我还没到那种举重若轻的层次吧,我觉得这真的是一个不错的提升自我修养的手段。(当然,即便是现在回想,也觉得这过程中犯傻不断,给我一个重来的机会,绝对不选这么大话题动笔)</p>
<p>年中,参与 Perl 中国社区大会的举办。和其他老手不同,我本身只在三年前听过一次 Perl Beijing Workshop 而已,这次直接就被“骗上贼船”,作为报名网站的管理员维护一点信息发布,邀请还不算很熟的朋友一同演讲,当然也贡献了自己的第一次公开演讲。演讲前一晚,特意在家试讲让老婆帮忙提意见,毫不意外的被老婆批为乱七八糟。最后临场刚好卡在45分钟结束,但是从反响来看,依然选题有些宽泛,要在一个演讲里同时展示 Elasticsearch 的知识、logstash 的知识、Message::Passing 的知识,只能让听众更加迷惑。意外之喜是这次演讲的 slide 后来发到网上,倒是被不少外国人 like 甚至转到 twitter 上,赢得不少关注。</p>
<p>原本在 ChinaUnix 论坛上答应在大会的 lighting talk 上稍微讲一下 autobox 的运用,结果有事提前退场了,感觉失约这种事情真是超级不好意思,但愿明年还有 Perl Workshop 来给我弥补!</p>
<p>会上见到了 90 后的 Perler,会后没多久读到 stevan little 收回他《Perl isn’t dead, Perl is dead end》一文并重启 MOP 计划的通告,让我对 Perl 的未来依然有信心多了。</p>
<p>话题之外,参加了 RubyConfChina2013,Rubist 普遍比 Perl Monger 土豪多了。我们演讲人清一色小黑,他们清一色苹果……</p>
<p>年底读了许式伟的《Go 语言编程》,完全不是给我们这些非科班的运维人员读的东西,看完以后一点对 golang 的兴趣都没有增加,虽然本人依然坚持“能够在运维社区火起来的东西肯定是比较靠谱的”。</p>
<p>博客方面,欠了两篇一直没时间写,关于 docker 如何自己作image,以及 staticperl 的使用。</p>
<p>工作之外,花了点时间在一些开源社区:</p>
<ol>
<li>
<p>logstash</p>
<p>logstash 在今年渐成气候,连它的竞争对手 fluentd 在年度报告中都承认 logstash 在美国已经势不可挡(fluentd的主阵地是日本)。个人在 logstash 代码方面只是保持跟踪阅读,因为没有业务需求推动,所以不再跟去年那样大肆修改代码。倒是通过weibo、QQ等方式回答了应该有好几十个人的问题,最后在各方鼓励下开设了 logstash 的QQ群(315428175),欢迎爱好者加入~</p>
<p>另一个惊喜的事情,两位 logstash 同好在问完问题之后,主动送了《Elasticsearch Server》和《thelogstashbook》给我。</p>
</li>
<li>
<p>Rexify</p>
<p>Rexify 是德国2012年度最佳开源软件。不过受国内 Perl 社区总体不给力的影响,不可能如 Python 的 SaltStack 那样突然窜起。去年提交的 krb5 认证的 patch 在今年终于被作者合并,年中翻译完成了 <a href="http://rex.perl-china.com">Rexify 中文站</a> 后,有一段时间没有进展了。</p>
</li>
<li>
<p>Docker</p>
<p>这是今年下半年重点看好的项目。博客中从 8 月开始就有好几篇关于这个内容,甚至专门订阅了 DockerWeekly。Docker 文档非常全,使用非常简单,实在爱不释手,最近老琢磨如何用在工作中去。最后这周,配合 pstuifzand 改进 Docker 的 <a href="https://github.com/chenryn/docker-perl">Perl 客户端</a>,主要是他写的时候 docker 默认还是监听在本地的 4243 端口,现在已经改成 <code class="highlighter-rouge">/var/run/docker.sock</code> 了。于是把 <code class="highlighter-rouge">Net::Docker</code> 里 <code class="highlighter-rouge">LWP::UserAgent</code> 和 <code class="highlighter-rouge">AnyEvent::HTTP</code> 的 Unix-Socket 支持都实现出来。</p>
<p>另一方面,也给 Rexify 项目实现了 <code class="highlighter-rouge">Rex::Virtualization::Docker</code> 的支持,不过这个则是调用 docker 命令行的方式。</p>
<p>最后,强烈谴责 GFW 屏蔽 Docker 的 CDN 边缘节点 IP 地址的行为。我本来提议让官方提供镜像方式,让我们在国内作镜像服务。结果官方表示 <code class="highlighter-rouge">docker pull</code> 的过程中连接了多个 api 服务,不是单单搞镜像可以解决的。目前只能是通过绑定 /etc/hosts 的方式直接访问官方的源站 IP,不走 CDN 了。</p>
</li>
<li>
<p>Perl</p>
<p>Perl5 社区今年发起了一系列活动,激发社区活跃性。其中包括一项挽救濒死模块。根据自己的情况,花了 3 个月时间走流程,最终成功认领了 <code class="highlighter-rouge">HTML::TagHelper</code> 模块。这个模块的原作者是个北欧妹纸,后来写 php 去了,看 linkedin 信息都已经是 CTO 了。</p>
<p>因为工作中用到 MooseFS,所以仿造 moosefs.cgi 里的接口写了 Perl 版的模块发到 CPAN,结果 moosefs 的作者之一 peter aNeutrino 主动发邮件来问是否需要更多帮助。只能说大神们真的好热情……</p>
<p>和氓氓等一起试图给 PerlChina 增加活跃气氛,申请了 <a href="http://weibo.com/perldaily">@perldaily</a> 帐号专门发技术内容,创建了 <a href="http://www.perl-china.com">www.perl-china.com</a> 网站。总的来说,努力过,成效就不在能力范围内了。</p>
</li>
</ol>
<p>再说一些跟国外的事情,虽然都是小事,但迈出第一步,总是值得纪念的:</p>
<ol>
<li>Perl 社区每年12月会发起 advent calendar 活动。今年主动去日本的 Qiita 技术社区投稿,写了一篇文章讲 Rex::Box 的运用。虽然写不来日语,不过代码就是最好的语言~~</li>
<li>被 facebook 的 tech recruiter 找上来聊天,算是见识了一下除了美剧以外的英语,嗯,也就如此了。</li>
</ol>
<p>最后一项不得不记的,关于比特币。我个人其实没有资金投入到这场狂欢中,不过暴赚几十万的同事、步步踩空的同事历历在目。一方面也回顾了一下当初的股票分析知识,通过程序作出了一些“不负责任的”指导意见,没有被同事们暴揍,算是一件很值得娱乐的事情~<br />
从严肃意义上来说,这件事情提醒了已经迈向26岁的自己,你已经不年轻了,Perl 之外,请考虑人生和理财的问题。</p>
<p>“Yes, sir!”</p>
为比特币绘制 MACD、BOLL、KDJ 指标图
2013-12-09T00:00:00+08:00
python
highcharts
bitcoin
http://chenlinux.com/2013/12/09/draw-charts-of-bitcoin
<p>比特币是最近相当火爆的一个金融衍生品(瞧咱这口径)。比特币中国提供了一系列 API 来获取和操纵其市场内的比特币。我的小伙伴们基于其 API,完成了一套交易程序。为了提高操作的有效性和技术性,同时作为 python 学习需要,我也参与进来,仿造股票交易软件,为比特币中国绘制了一系列指标图,包括 MACD、BOLL、KDJ 等。截止上周,btc123 也开始提供了 MACD 指标图,所以把自己的实现贴到博客。</p>
<p>首先是获取数据,比特币中国的 API 是个很鬼怪的东西,实时交易数据的接口,返回的数据中最高最低和成交量都是基于过去24小时的,要知道比特币交易是没有休市的啊。所以获取数据过程中需要自己计算这些。这里考虑到股市一般一天实际交易4小时,所以整个设计也是默认4小时的图形展示。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#!/usr/bin/env python3</span>
<span class="c"># -*- coding: utf-8 -*-</span>
<span class="c"># query price data from BTCChina.</span>
<span class="kn">from</span> <span class="nn">urllib</span> <span class="kn">import</span> <span class="n">urlopen</span>
<span class="kn">from</span> <span class="nn">ast</span> <span class="kn">import</span> <span class="n">literal_eval</span>
<span class="kn">import</span> <span class="nn">MySQLdb</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">yaml</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="n">config</span> <span class="o">=</span> <span class="n">yaml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="nb">open</span><span class="p">(</span><span class="s">'config.yaml'</span><span class="p">))</span>
<span class="n">conn</span> <span class="o">=</span> <span class="n">MySQLdb</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'host'</span><span class="p">],</span><span class="n">user</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'username'</span><span class="p">],</span><span class="n">passwd</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'password'</span><span class="p">],</span><span class="n">db</span> <span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'databasename'</span><span class="p">],</span><span class="n">charset</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'encoding'</span><span class="p">]</span> <span class="p">)</span>
<span class="k">def</span> <span class="nf">write_db</span><span class="p">(</span><span class="n">datas</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">cur_write</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
<span class="n">sql</span> <span class="o">=</span> <span class="s">"insert into ticker(sell, buy, last, vol, high, low) values( </span><span class="si">%</span><span class="s">s, </span><span class="si">%</span><span class="s">s, </span><span class="si">%</span><span class="s">s,</span><span class="si">%</span><span class="s">s,</span><span class="si">%</span><span class="s">s,</span><span class="si">%</span><span class="s">s)"</span>
<span class="n">cur_write</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span><span class="n">datas</span><span class="p">)</span>
<span class="n">conn</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
<span class="n">cur_write</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">except</span> <span class="n">MySQLdb</span><span class="o">.</span><span class="n">Error</span><span class="p">,</span><span class="n">e</span><span class="p">:</span>
<span class="k">print</span> <span class="s">"Mysql error </span><span class="si">%</span><span class="s">d : </span><span class="si">%</span><span class="s">s."</span> <span class="o">%</span> <span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">args</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">e</span><span class="o">.</span><span class="n">args</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="k">def</span> <span class="nf">get_tid</span><span class="p">():</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">vol_url</span> <span class="o">=</span> <span class="n">config</span><span class="p">[</span><span class="s">'btcchina'</span><span class="p">][</span><span class="s">'vol_url'</span><span class="p">]</span>
<span class="n">remote_file</span> <span class="o">=</span> <span class="n">urlopen</span><span class="p">(</span><span class="n">vol_url</span><span class="p">)</span>
<span class="n">remote_data</span> <span class="o">=</span> <span class="n">remote_file</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="n">remote_file</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="n">remote_data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">remote_data</span><span class="p">))</span>
<span class="k">return</span> <span class="n">remote_data</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="s">'tid'</span><span class="p">]</span>
<span class="k">except</span> <span class="n">MySQLdb</span><span class="o">.</span><span class="n">Error</span><span class="p">,</span><span class="n">e</span><span class="p">:</span>
<span class="k">print</span> <span class="s">"Mysql error </span><span class="si">%</span><span class="s">d : </span><span class="si">%</span><span class="s">s."</span> <span class="o">%</span> <span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">args</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">e</span><span class="o">.</span><span class="n">args</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="k">def</span> <span class="nf">get_ohlc</span><span class="p">(</span><span class="n">num</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">read</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
<span class="n">hlvsql</span> <span class="o">=</span> <span class="s">"select max(last),min(last) from ticker where time between date_add(now(),interval -</span><span class="si">%</span><span class="s">s minute) and now()"</span> <span class="o">%</span> <span class="n">num</span>
<span class="n">read</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">hlvsql</span><span class="p">)</span>
<span class="n">high</span><span class="p">,</span> <span class="n">low</span> <span class="o">=</span> <span class="n">read</span><span class="o">.</span><span class="n">fetchone</span><span class="p">()</span>
<span class="n">closesql</span> <span class="o">=</span> <span class="s">"select last from ticker where time between date_add(now(),interval -</span><span class="si">%</span><span class="s">s minute) and now() order by time desc limit 1"</span> <span class="o">%</span> <span class="n">num</span>
<span class="n">read</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">closesql</span><span class="p">)</span>
<span class="n">close</span> <span class="o">=</span> <span class="n">read</span><span class="o">.</span><span class="n">fetchone</span><span class="p">()</span>
<span class="n">opensql</span> <span class="o">=</span> <span class="s">"select last from ticker where time between date_add(now(),interval -</span><span class="si">%</span><span class="s">s minute) and now() order by time asc limit 1"</span> <span class="o">%</span> <span class="n">num</span>
<span class="n">read</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">opensql</span><span class="p">)</span>
<span class="n">opend</span> <span class="o">=</span> <span class="n">read</span><span class="o">.</span><span class="n">fetchone</span><span class="p">()</span>
<span class="k">return</span> <span class="n">opend</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">high</span><span class="p">,</span> <span class="n">low</span><span class="p">,</span> <span class="n">close</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">except</span> <span class="n">MySQLdb</span><span class="o">.</span><span class="n">Error</span><span class="p">,</span><span class="n">e</span><span class="p">:</span>
<span class="k">print</span> <span class="s">"Mysql error </span><span class="si">%</span><span class="s">d : </span><span class="si">%</span><span class="s">s."</span> <span class="o">%</span> <span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">args</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">e</span><span class="o">.</span><span class="n">args</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="k">def</span> <span class="nf">write_ohlc</span><span class="p">(</span><span class="n">data</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">cur_write</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
<span class="n">ohlcsql</span> <span class="o">=</span> <span class="s">'insert into ohlc(open, high, low, close, vol) values( </span><span class="si">%</span><span class="s">s, </span><span class="si">%</span><span class="s">s, </span><span class="si">%</span><span class="s">s, </span><span class="si">%</span><span class="s">s, </span><span class="si">%</span><span class="s">s)'</span>
<span class="n">cur_write</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">ohlcsql</span><span class="p">,</span> <span class="n">data</span><span class="p">)</span>
<span class="n">conn</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
<span class="n">cur_write</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">except</span> <span class="n">MySQLdb</span><span class="o">.</span><span class="n">Error</span><span class="p">,</span><span class="n">e</span><span class="p">:</span>
<span class="k">print</span> <span class="s">"Mysql error </span><span class="si">%</span><span class="s">d : </span><span class="si">%</span><span class="s">s."</span> <span class="o">%</span> <span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">args</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">e</span><span class="o">.</span><span class="n">args</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="k">except</span> <span class="nb">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"执行Mysql写入数据时出错: </span><span class="si">%</span><span class="s">s"</span> <span class="o">%</span> <span class="n">e</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">instance</span><span class="p">():</span>
<span class="k">try</span><span class="p">:</span>
<span class="c"># returns something like {"high":738.88,"low":689.10,"buy":713.50,"sell":717.30,"last":717.41,"vol":4797.32000000}</span>
<span class="n">remote_file</span> <span class="o">=</span> <span class="n">urlopen</span><span class="p">(</span><span class="n">config</span><span class="p">[</span><span class="s">'btcchina'</span><span class="p">][</span><span class="s">'ticker_url'</span><span class="p">])</span>
<span class="n">remote_data</span> <span class="o">=</span> <span class="n">remote_file</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="n">remote_file</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="n">remote_data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">remote_data</span><span class="p">))[</span><span class="s">'ticker'</span><span class="p">]</span>
<span class="c"># remote_data = {key:literal_eval(remote_data[key]) for key in remote_data}</span>
<span class="k">except</span><span class="p">:</span>
<span class="n">remote_data</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">datas</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">remote_data</span><span class="p">:</span>
<span class="n">datas</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">remote_data</span><span class="p">[</span><span class="n">key</span><span class="p">])</span>
<span class="k">return</span> <span class="n">datas</span>
<span class="n">lastid</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">ohlc_period</span> <span class="o">=</span> <span class="mi">60</span>
<span class="n">next_ohlc</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">())</span> <span class="o">/</span> <span class="n">ohlc_period</span> <span class="o">*</span> <span class="n">ohlc_period</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">datas</span> <span class="o">=</span> <span class="n">instance</span><span class="p">()</span>
<span class="k">if</span> <span class="n">datas</span><span class="p">:</span>
<span class="n">write_db</span><span class="p">(</span><span class="n">datas</span><span class="p">)</span>
<span class="k">if</span><span class="p">(</span><span class="nb">int</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">())</span> <span class="o">></span> <span class="n">next_ohlc</span><span class="p">):</span>
<span class="n">next_ohlc</span> <span class="o">+=</span> <span class="n">ohlc_period</span>
<span class="n">data</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">get_ohlc</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
<span class="n">latestid</span> <span class="o">=</span> <span class="n">get_tid</span><span class="p">()</span>
<span class="n">data</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">int</span><span class="p">(</span><span class="n">latestid</span><span class="p">)</span> <span class="o">-</span> <span class="nb">int</span><span class="p">(</span><span class="n">lastid</span><span class="p">))</span>
<span class="n">lastid</span> <span class="o">=</span> <span class="n">latestid</span>
<span class="n">write_ohlc</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</code></pre>
</div>
<p>这里主要把实时数据存入ticker表,分钟统计数据存入ohlc表。然后是各指标算法。首先是 MACD :</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#/*******************************************************************************</span>
<span class="c"># * Author: Chenlin Rao | Renren inc.</span>
<span class="c"># * Email: rao.chenlin@gmail.com</span>
<span class="c"># * Last modified: 2013-11-26 22:02</span>
<span class="c"># * Filename: macd.py</span>
<span class="c"># * Description: </span>
<span class="c"># EMA(12)=LastEMA(12)* 11/13 + Close * 2/13</span>
<span class="c"># EMA(26)=LastEMA(26)* 25/27 + Close * 2/27</span>
<span class="c"># </span>
<span class="c"># DIF=EMA(12)-EMA(26)</span>
<span class="c"># DEA=LastDEA * 8/10 + DIF * 2/10</span>
<span class="c"># MACD=(DIF-DEA) * 2</span>
<span class="c"># * *****************************************************************************/</span>
<span class="c">#!/usr/bin/env python</span>
<span class="c"># -*- coding: utf-8 -*-</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">hashlib</span>
<span class="kn">import</span> <span class="nn">MySQLdb</span>
<span class="kn">import</span> <span class="nn">yaml</span>
<span class="k">class</span> <span class="nc">MACD</span><span class="p">():</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">config</span> <span class="o">=</span> <span class="n">yaml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="nb">open</span><span class="p">(</span><span class="s">'config.yml'</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sleep_time</span> <span class="o">=</span> <span class="n">config</span><span class="p">[</span><span class="s">'btcchina'</span><span class="p">][</span><span class="s">'trade_option'</span><span class="p">][</span><span class="s">'sleep_time'</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">conn</span> <span class="o">=</span> <span class="n">MySQLdb</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'host'</span><span class="p">],</span><span class="n">user</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'username'</span><span class="p">],</span><span class="n">passwd</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'password'</span><span class="p">],</span><span class="n">db</span> <span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'databasename'</span><span class="p">],</span><span class="n">charset</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'encoding'</span><span class="p">]</span> <span class="p">)</span>
<span class="k">def</span> <span class="nf">_getclose</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">num</span><span class="p">):</span>
<span class="n">read</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
<span class="n">sql</span> <span class="o">=</span> <span class="s">"select close,time from ohlc order by id desc limit </span><span class="si">%</span><span class="s">s"</span> <span class="o">%</span> <span class="n">num</span>
<span class="n">count</span> <span class="o">=</span> <span class="n">read</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">)</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">read</span><span class="o">.</span><span class="n">fetchall</span><span class="p">()</span>
<span class="k">return</span> <span class="n">results</span><span class="p">[::</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">_ema</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">s</span><span class="p">,</span> <span class="n">n</span><span class="p">):</span>
<span class="s">"""
returns an n period exponential moving average for
the time series s
s is a list ordered from oldest (index 0) to most
recent (index -1)
n is an integer
returns a numeric array of the exponential
moving average
"""</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="o"><=</span> <span class="n">n</span><span class="p">:</span>
<span class="k">return</span> <span class="s">"No enough item in </span><span class="si">%</span><span class="s">s"</span> <span class="o">%</span> <span class="n">s</span>
<span class="n">ema</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">j</span> <span class="o">=</span> <span class="mi">1</span>
<span class="c">#get n sma first and calculate the next n period ema</span>
<span class="n">sma</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">(</span><span class="n">s</span><span class="p">[:</span><span class="n">n</span><span class="p">])</span> <span class="o">/</span> <span class="n">n</span>
<span class="n">multiplier</span> <span class="o">=</span> <span class="mi">2</span> <span class="o">/</span> <span class="nb">float</span><span class="p">(</span><span class="mi">1</span> <span class="o">+</span> <span class="n">n</span><span class="p">)</span>
<span class="n">ema</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">sma</span><span class="p">)</span>
<span class="c">#EMA(current) = ( (Price(current) - EMA(prev) ) x Multiplier) + EMA(prev)</span>
<span class="n">ema</span><span class="o">.</span><span class="n">append</span><span class="p">((</span> <span class="p">(</span><span class="n">s</span><span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="o">-</span> <span class="n">sma</span><span class="p">)</span> <span class="o">*</span> <span class="n">multiplier</span><span class="p">)</span> <span class="o">+</span> <span class="n">sma</span><span class="p">)</span>
<span class="c">#now calculate the rest of the values</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">s</span><span class="p">[</span><span class="n">n</span><span class="o">+</span><span class="mi">1</span><span class="p">:]:</span>
<span class="n">tmp</span> <span class="o">=</span> <span class="p">(</span> <span class="p">(</span><span class="n">i</span> <span class="o">-</span> <span class="n">ema</span><span class="p">[</span><span class="n">j</span><span class="p">])</span> <span class="o">*</span> <span class="n">multiplier</span><span class="p">)</span> <span class="o">+</span> <span class="n">ema</span><span class="p">[</span><span class="n">j</span><span class="p">]</span>
<span class="n">j</span> <span class="o">=</span> <span class="n">j</span> <span class="o">+</span> <span class="mi">1</span>
<span class="n">ema</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">tmp</span><span class="p">)</span>
<span class="k">return</span> <span class="n">ema</span>
<span class="k">def</span> <span class="nf">getMACD</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">n</span><span class="p">):</span>
<span class="n">array</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_getclose</span><span class="p">(</span><span class="n">n</span><span class="p">)</span>
<span class="n">prices</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">array</span><span class="p">)</span>
<span class="n">t</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="nb">int</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">mktime</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">timetuple</span><span class="p">()))</span> <span class="o">*</span> <span class="mi">1000</span><span class="p">,</span> <span class="n">array</span><span class="p">)</span>
<span class="n">short_ema</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_ema</span><span class="p">(</span><span class="n">prices</span><span class="p">,</span> <span class="mi">12</span><span class="p">)</span>
<span class="n">long_ema</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_ema</span><span class="p">(</span><span class="n">prices</span><span class="p">,</span> <span class="mi">26</span><span class="p">)</span>
<span class="n">diff</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">-</span><span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="nb">zip</span><span class="p">(</span><span class="n">short_ema</span><span class="p">[::</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="n">long_ema</span><span class="p">[::</span><span class="o">-</span><span class="mi">1</span><span class="p">]))</span>
<span class="n">diff</span><span class="o">.</span><span class="n">reverse</span><span class="p">()</span>
<span class="n">dea</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_ema</span><span class="p">(</span><span class="n">diff</span><span class="p">,</span> <span class="mi">9</span><span class="p">)</span>
<span class="n">bar</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="mi">2</span><span class="o">*</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">-</span><span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">]),</span> <span class="nb">zip</span><span class="p">(</span><span class="n">diff</span><span class="p">[::</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="n">dea</span><span class="p">[::</span><span class="o">-</span><span class="mi">1</span><span class="p">]))</span>
<span class="n">bar</span><span class="o">.</span><span class="n">reverse</span><span class="p">()</span>
<span class="k">return</span> <span class="nb">zip</span><span class="p">(</span><span class="n">t</span><span class="p">[</span><span class="mi">33</span><span class="p">:],</span> <span class="n">diff</span><span class="p">[</span><span class="mi">8</span><span class="p">:]),</span> <span class="nb">zip</span><span class="p">(</span><span class="n">t</span><span class="p">[</span><span class="mi">33</span><span class="p">:],</span> <span class="n">dea</span><span class="p">),</span> <span class="nb">zip</span><span class="p">(</span><span class="n">t</span><span class="p">[</span><span class="mi">33</span><span class="p">:],</span> <span class="n">bar</span><span class="p">)</span>
</code></pre>
</div>
<p>然后是 BOLL :</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#/*******************************************************************************</span>
<span class="c"># * Author: Chenlin Rao | Renren inc.</span>
<span class="c"># * Email: rao.chenlin@gmail.com</span>
<span class="c"># * Last modified: 2013-11-26 22:02</span>
<span class="c"># * Filename: macd.py</span>
<span class="c"># * Description: </span>
<span class="c"># MA=avg(close(20))</span>
<span class="c"># MD=std(close(20))</span>
<span class="c"># </span>
<span class="c"># MB=MA(20)</span>
<span class="c"># UP=MB + 2*MD</span>
<span class="c"># DN=MB - 2*MD</span>
<span class="c"># * *****************************************************************************/</span>
<span class="c">#!/usr/bin/env python</span>
<span class="c"># -*- coding: utf-8 -*-</span>
<span class="kn">import</span> <span class="nn">random</span>
<span class="kn">import</span> <span class="nn">hashlib</span>
<span class="kn">import</span> <span class="nn">MySQLdb</span>
<span class="kn">import</span> <span class="nn">yaml</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="k">class</span> <span class="nc">BOLL</span><span class="p">():</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">config</span> <span class="o">=</span> <span class="n">yaml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="nb">open</span><span class="p">(</span><span class="s">'config.yml'</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sleep_time</span> <span class="o">=</span> <span class="n">config</span><span class="p">[</span><span class="s">'btcchina'</span><span class="p">][</span><span class="s">'trade_option'</span><span class="p">][</span><span class="s">'sleep_time'</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">conn</span> <span class="o">=</span> <span class="n">MySQLdb</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'host'</span><span class="p">],</span><span class="n">user</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'username'</span><span class="p">],</span><span class="n">passwd</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'password'</span><span class="p">],</span><span class="n">db</span> <span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'databasename'</span><span class="p">],</span><span class="n">charset</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'encoding'</span><span class="p">]</span> <span class="p">)</span>
<span class="k">def</span> <span class="nf">_getMA</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">array</span><span class="p">):</span>
<span class="n">length</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">array</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">sum</span><span class="p">(</span><span class="n">array</span><span class="p">)</span> <span class="o">/</span> <span class="n">length</span>
<span class="k">def</span> <span class="nf">_getMD</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">array</span><span class="p">):</span>
<span class="n">length</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">array</span><span class="p">)</span>
<span class="n">average</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">(</span><span class="n">array</span><span class="p">)</span> <span class="o">/</span> <span class="n">length</span>
<span class="n">d</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">array</span><span class="p">:</span> <span class="n">d</span> <span class="o">+=</span> <span class="p">(</span><span class="n">i</span> <span class="o">-</span> <span class="n">average</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span>
<span class="k">return</span> <span class="p">(</span><span class="n">d</span><span class="o">/</span><span class="n">length</span><span class="p">)</span> <span class="o">**</span> <span class="mf">0.5</span>
<span class="k">def</span> <span class="nf">getOHLC</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">num</span><span class="p">):</span>
<span class="n">read</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
<span class="n">sql</span> <span class="o">=</span> <span class="s">"select time,open,high,low,close,vol from ohlc order by id desc limit </span><span class="si">%</span><span class="s">s"</span> <span class="o">%</span> <span class="n">num</span>
<span class="n">count</span> <span class="o">=</span> <span class="n">read</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">)</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">read</span><span class="o">.</span><span class="n">fetchall</span><span class="p">()</span>
<span class="k">return</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="p">[</span><span class="nb">int</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">mktime</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">timetuple</span><span class="p">()))</span> <span class="o">*</span> <span class="mi">1000</span><span class="p">,</span> <span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span><span class="n">x</span><span class="p">[</span><span class="mi">2</span><span class="p">],</span><span class="n">x</span><span class="p">[</span><span class="mi">3</span><span class="p">],</span><span class="n">x</span><span class="p">[</span><span class="mi">4</span><span class="p">],</span><span class="n">x</span><span class="p">[</span><span class="mi">5</span><span class="p">]],</span> <span class="n">results</span><span class="p">[::</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
<span class="k">def</span> <span class="nf">_getCur</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">fromtime</span><span class="p">):</span>
<span class="n">curread</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
<span class="n">cursql</span> <span class="o">=</span> <span class="s">"select last,vol from ticker where time between date_add('</span><span class="si">%</span><span class="s">s', interval -0 minute) and now()"</span> <span class="o">%</span> <span class="n">time</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s">'</span><span class="si">%</span><span class="s">F </span><span class="si">%</span><span class="s">T'</span><span class="p">,</span> <span class="n">time</span><span class="o">.</span><span class="n">localtime</span><span class="p">(</span><span class="n">fromtime</span><span class="p">))</span>
<span class="n">curread</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">cursql</span><span class="p">)</span>
<span class="n">curlist</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">curread</span><span class="o">.</span><span class="n">fetchall</span><span class="p">())</span>
<span class="n">vollist</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">curread</span><span class="o">.</span><span class="n">fetchall</span><span class="p">())</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">curlist</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">int</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">())</span><span class="o">*</span><span class="mi">1000</span><span class="p">,</span> <span class="n">curlist</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="nb">max</span><span class="p">(</span><span class="n">curlist</span><span class="p">),</span> <span class="nb">min</span><span class="p">(</span><span class="n">curlist</span><span class="p">),</span> <span class="n">curlist</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="nb">sum</span><span class="p">(</span><span class="n">vollist</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">None</span>
<span class="k">def</span> <span class="nf">_getClose</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">matrix</span><span class="p">):</span>
<span class="n">close</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="mi">4</span><span class="p">],</span> <span class="n">matrix</span><span class="p">)</span>
<span class="k">return</span> <span class="n">close</span>
<span class="k">def</span> <span class="nf">getBOLL</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">num</span><span class="p">,</span> <span class="n">days</span><span class="p">):</span>
<span class="n">matrix</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">getOHLC</span><span class="p">(</span><span class="n">num</span><span class="p">)</span>
<span class="n">cur</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_getCur</span><span class="p">(</span><span class="n">matrix</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span><span class="o">/</span><span class="mi">1000</span><span class="p">)</span>
<span class="k">if</span> <span class="n">cur</span><span class="p">:</span>
<span class="n">matrix</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">cur</span><span class="p">)</span>
<span class="n">array</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_getClose</span><span class="p">(</span><span class="n">matrix</span><span class="p">)</span>
<span class="n">up</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">mb</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">dn</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">days</span>
<span class="k">while</span> <span class="n">x</span> <span class="o"><</span> <span class="nb">len</span><span class="p">(</span><span class="n">array</span><span class="p">):</span>
<span class="n">curmb</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_getMA</span><span class="p">(</span><span class="n">array</span><span class="p">[</span><span class="n">x</span><span class="o">-</span><span class="n">days</span><span class="p">:</span><span class="n">x</span><span class="p">])</span>
<span class="n">curmd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_getMD</span><span class="p">(</span><span class="n">array</span><span class="p">[</span><span class="n">x</span><span class="o">-</span><span class="n">days</span><span class="p">:</span><span class="n">x</span><span class="p">])</span>
<span class="n">mb</span><span class="o">.</span><span class="n">append</span><span class="p">(</span> <span class="p">[</span> <span class="n">matrix</span><span class="p">[</span><span class="n">x</span><span class="p">][</span><span class="mi">0</span><span class="p">],</span> <span class="n">curmb</span> <span class="p">]</span> <span class="p">)</span>
<span class="n">up</span><span class="o">.</span><span class="n">append</span><span class="p">(</span> <span class="p">[</span> <span class="n">matrix</span><span class="p">[</span><span class="n">x</span><span class="p">][</span><span class="mi">0</span><span class="p">],</span> <span class="n">curmb</span> <span class="o">+</span> <span class="mi">2</span> <span class="o">*</span> <span class="n">curmd</span> <span class="p">]</span> <span class="p">)</span>
<span class="n">dn</span><span class="o">.</span><span class="n">append</span><span class="p">(</span> <span class="p">[</span> <span class="n">matrix</span><span class="p">[</span><span class="n">x</span><span class="p">][</span><span class="mi">0</span><span class="p">],</span> <span class="n">curmb</span> <span class="o">-</span> <span class="mi">2</span> <span class="o">*</span> <span class="n">curmd</span> <span class="p">]</span> <span class="p">)</span>
<span class="n">x</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="n">matrix</span><span class="p">[</span><span class="n">days</span><span class="p">:],</span> <span class="n">up</span><span class="p">,</span> <span class="n">mb</span><span class="p">,</span> <span class="n">dn</span>
</code></pre>
</div>
<p>最后是 KDJ :</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#/*******************************************************************************</span>
<span class="c"># * Author: Chenlin Rao | Renren inc.</span>
<span class="c"># * Email: rao.chenlin@gmail.com</span>
<span class="c"># * Last modified: 2013-11-26 22:02</span>
<span class="c"># * Filename: macd.py</span>
<span class="c"># * Description: </span>
<span class="c"># RSV=(close-low(9))/(high(9)-low(9))*100</span>
<span class="c"># K=SMA(RSV(3), 1)</span>
<span class="c"># D=SMA(K(3), 1)</span>
<span class="c"># J=3*K-2*D</span>
<span class="c"># * *****************************************************************************/</span>
<span class="c">#!/usr/bin/env python</span>
<span class="c"># -*- coding: utf-8 -*-</span>
<span class="kn">import</span> <span class="nn">hashlib</span>
<span class="kn">import</span> <span class="nn">MySQLdb</span>
<span class="kn">import</span> <span class="nn">yaml</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="k">class</span> <span class="nc">KDJ</span><span class="p">():</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">config</span> <span class="o">=</span> <span class="n">yaml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="nb">open</span><span class="p">(</span><span class="s">'config.yml'</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sleep_time</span> <span class="o">=</span> <span class="n">config</span><span class="p">[</span><span class="s">'btcchina'</span><span class="p">][</span><span class="s">'trade_option'</span><span class="p">][</span><span class="s">'sleep_time'</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">conn</span> <span class="o">=</span> <span class="n">MySQLdb</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'host'</span><span class="p">],</span><span class="n">user</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'username'</span><span class="p">],</span><span class="n">passwd</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'password'</span><span class="p">],</span><span class="n">db</span> <span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'databasename'</span><span class="p">],</span><span class="n">charset</span><span class="o">=</span><span class="n">config</span><span class="p">[</span><span class="s">'database'</span><span class="p">][</span><span class="s">'encoding'</span><span class="p">]</span> <span class="p">)</span>
<span class="k">def</span> <span class="nf">_getHLC</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">num</span><span class="p">):</span>
<span class="n">read</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
<span class="n">sql</span> <span class="o">=</span> <span class="s">"select high,low,close,time from ohlc order by id desc limit </span><span class="si">%</span><span class="s">s"</span> <span class="o">%</span> <span class="n">num</span>
<span class="n">count</span> <span class="o">=</span> <span class="n">read</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">)</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">read</span><span class="o">.</span><span class="n">fetchall</span><span class="p">()</span>
<span class="k">return</span> <span class="n">results</span><span class="p">[::</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">_avg</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">a</span><span class="p">):</span>
<span class="n">length</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">sum</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="o">/</span> <span class="n">length</span>
<span class="k">def</span> <span class="nf">_getMA</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">values</span><span class="p">,</span> <span class="n">window</span><span class="p">):</span>
<span class="n">array</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">window</span>
<span class="k">while</span> <span class="n">x</span> <span class="o"><</span> <span class="nb">len</span><span class="p">(</span><span class="n">values</span><span class="p">):</span>
<span class="n">curmb</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_avg</span><span class="p">(</span><span class="n">values</span><span class="p">[</span><span class="n">x</span><span class="o">-</span><span class="n">window</span><span class="p">:</span><span class="n">x</span><span class="p">])</span>
<span class="n">array</span><span class="o">.</span><span class="n">append</span><span class="p">(</span> <span class="n">curmb</span> <span class="p">)</span>
<span class="n">x</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="n">array</span>
<span class="k">def</span> <span class="nf">_getRSV</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">arrays</span><span class="p">):</span>
<span class="n">rsv</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">times</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">x</span> <span class="o">=</span> <span class="mi">9</span>
<span class="k">while</span> <span class="n">x</span> <span class="o"><</span> <span class="nb">len</span><span class="p">(</span><span class="n">arrays</span><span class="p">):</span>
<span class="n">high</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">arrays</span><span class="p">[</span><span class="n">x</span><span class="o">-</span><span class="mi">9</span><span class="p">:</span><span class="n">x</span><span class="p">]))</span>
<span class="n">low</span> <span class="o">=</span> <span class="nb">min</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">arrays</span><span class="p">[</span><span class="n">x</span><span class="o">-</span><span class="mi">9</span><span class="p">:</span><span class="n">x</span><span class="p">]))</span>
<span class="n">close</span> <span class="o">=</span> <span class="n">arrays</span><span class="p">[</span><span class="n">x</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="mi">2</span><span class="p">]</span>
<span class="n">rsv</span><span class="o">.</span><span class="n">append</span><span class="p">(</span> <span class="p">(</span><span class="n">close</span><span class="o">-</span><span class="n">low</span><span class="p">)</span><span class="o">/</span><span class="p">(</span><span class="n">high</span><span class="o">-</span><span class="n">low</span><span class="p">)</span><span class="o">*</span><span class="mi">100</span> <span class="p">)</span>
<span class="n">t</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">mktime</span><span class="p">(</span><span class="n">arrays</span><span class="p">[</span><span class="n">x</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">timetuple</span><span class="p">()))</span> <span class="o">*</span> <span class="mi">1000</span>
<span class="n">times</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">t</span><span class="p">)</span>
<span class="n">x</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="n">times</span><span class="p">,</span> <span class="n">rsv</span>
<span class="k">def</span> <span class="nf">getKDJ</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">num</span><span class="p">):</span>
<span class="n">hlc</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_getHLC</span><span class="p">(</span><span class="n">num</span><span class="p">)</span>
<span class="n">t</span><span class="p">,</span> <span class="n">rsv</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_getRSV</span><span class="p">(</span><span class="n">hlc</span><span class="p">)</span>
<span class="n">k</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_getMA</span><span class="p">(</span><span class="n">rsv</span><span class="p">,</span><span class="mi">3</span><span class="p">)</span>
<span class="n">d</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_getMA</span><span class="p">(</span><span class="n">k</span><span class="p">,</span><span class="mi">3</span><span class="p">)</span>
<span class="n">j</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="mi">3</span><span class="o">*</span><span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">-</span><span class="mi">2</span><span class="o">*</span><span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="nb">zip</span><span class="p">(</span><span class="n">k</span><span class="p">[</span><span class="mi">3</span><span class="p">:],</span> <span class="n">d</span><span class="p">))</span>
<span class="k">return</span> <span class="nb">zip</span><span class="p">(</span><span class="n">t</span><span class="p">[</span><span class="mi">2</span><span class="p">:],</span> <span class="n">k</span><span class="p">),</span> <span class="nb">zip</span><span class="p">(</span><span class="n">t</span><span class="p">[</span><span class="mi">5</span><span class="p">:],</span> <span class="n">d</span><span class="p">),</span> <span class="nb">zip</span><span class="p">(</span><span class="n">t</span><span class="p">[</span><span class="mi">5</span><span class="p">:],</span> <span class="n">j</span><span class="p">)</span>
</code></pre>
</div>
<p>最后通过一个简单的python web框架完成界面展示,这个叫 bottle.py 的框架是个单文件,相当方便。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#!/usr/bin/python</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">yaml</span>
<span class="kn">from</span> <span class="nn">macd</span> <span class="kn">import</span> <span class="n">MACD</span>
<span class="kn">from</span> <span class="nn">boll</span> <span class="kn">import</span> <span class="n">BOLL</span>
<span class="kn">from</span> <span class="nn">kdj</span> <span class="kn">import</span> <span class="n">KDJ</span>
<span class="kn">from</span> <span class="nn">bottle</span> <span class="kn">import</span> <span class="n">route</span><span class="p">,</span> <span class="n">run</span><span class="p">,</span> <span class="n">static_file</span><span class="p">,</span> <span class="n">redirect</span><span class="p">,</span> <span class="n">template</span>
<span class="n">config</span> <span class="o">=</span> <span class="n">yaml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="nb">open</span><span class="p">(</span><span class="s">'config.yml'</span><span class="p">))</span>
<span class="n">color</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'cn'</span><span class="p">:{</span><span class="s">'up'</span><span class="p">:</span><span class="s">'#ff0000'</span><span class="p">,</span><span class="s">'dn'</span><span class="p">:</span><span class="s">'#00ff00'</span><span class="p">},</span>
<span class="s">'us'</span><span class="p">:{</span><span class="s">'dn'</span><span class="p">:</span><span class="s">'#ff0000'</span><span class="p">,</span><span class="s">'up'</span><span class="p">:</span><span class="s">'#00ff00'</span><span class="p">},</span>
<span class="p">}</span>
<span class="nd">@route</span><span class="p">(</span><span class="s">'/'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">index</span><span class="p">():</span>
<span class="n">redirect</span><span class="p">(</span><span class="s">'/mkb/240'</span><span class="p">)</span>
<span class="nd">@route</span><span class="p">(</span><span class="s">'/mkb/<ago:int>'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">mkb</span><span class="p">(</span><span class="n">ago</span><span class="p">):</span>
<span class="n">like</span> <span class="o">=</span> <span class="n">config</span><span class="p">[</span><span class="s">'webui'</span><span class="p">][</span><span class="s">'color'</span><span class="p">]</span>
<span class="k">return</span> <span class="n">template</span><span class="p">(</span><span class="s">'webui'</span><span class="p">,</span> <span class="n">ago</span> <span class="o">=</span> <span class="n">ago</span><span class="p">,</span> <span class="n">color</span> <span class="o">=</span> <span class="n">color</span><span class="p">[</span><span class="n">like</span><span class="p">])</span>
<span class="nd">@route</span><span class="p">(</span><span class="s">'/js/<filename>'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">js</span><span class="p">(</span><span class="n">filename</span><span class="p">):</span>
<span class="k">return</span> <span class="n">static_file</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">root</span><span class="o">=</span><span class="s">'./js/'</span><span class="p">)</span>
<span class="nd">@route</span><span class="p">(</span><span class="s">'/boll'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">boll</span><span class="p">():</span>
<span class="k">return</span> <span class="s">"boll"</span>
<span class="nd">@route</span><span class="p">(</span><span class="s">'/macd/<day:int>'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">macd</span><span class="p">(</span><span class="n">day</span><span class="p">):</span>
<span class="n">m</span> <span class="o">=</span> <span class="n">MACD</span><span class="p">()</span>
<span class="n">dif</span><span class="p">,</span> <span class="n">dea</span><span class="p">,</span> <span class="n">bar</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">getMACD</span><span class="p">(</span><span class="n">day</span><span class="p">)</span>
<span class="k">return</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">({</span><span class="s">'dif'</span><span class="p">:</span><span class="n">dif</span><span class="p">,</span> <span class="s">'dea'</span><span class="p">:</span><span class="n">dea</span><span class="p">,</span> <span class="s">'bar'</span><span class="p">:</span><span class="n">bar</span><span class="p">})</span>
<span class="nd">@route</span><span class="p">(</span><span class="s">'/boll/<day:int>'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">boll</span><span class="p">(</span><span class="n">day</span><span class="p">):</span>
<span class="n">b</span> <span class="o">=</span> <span class="n">BOLL</span><span class="p">()</span>
<span class="n">ohlc</span><span class="p">,</span> <span class="n">up</span><span class="p">,</span> <span class="n">md</span><span class="p">,</span> <span class="n">dn</span> <span class="o">=</span> <span class="n">b</span><span class="o">.</span><span class="n">getBOLL</span><span class="p">(</span><span class="n">day</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span>
<span class="k">return</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">({</span><span class="s">'ohlc'</span><span class="p">:</span><span class="n">ohlc</span><span class="p">,</span> <span class="s">'up'</span><span class="p">:</span><span class="n">up</span><span class="p">,</span> <span class="s">'md'</span><span class="p">:</span><span class="n">md</span><span class="p">,</span> <span class="s">'dn'</span><span class="p">:</span><span class="n">dn</span><span class="p">})</span>
<span class="nd">@route</span><span class="p">(</span><span class="s">'/kdj/<day:int>'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">kdj</span><span class="p">(</span><span class="n">day</span><span class="p">):</span>
<span class="n">kdj</span> <span class="o">=</span> <span class="n">KDJ</span><span class="p">()</span>
<span class="n">k</span><span class="p">,</span> <span class="n">d</span><span class="p">,</span> <span class="n">j</span> <span class="o">=</span> <span class="n">kdj</span><span class="o">.</span><span class="n">getKDJ</span><span class="p">(</span><span class="n">day</span><span class="p">)</span>
<span class="k">return</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">({</span><span class="s">'k'</span><span class="p">:</span><span class="n">k</span><span class="p">,</span> <span class="s">'d'</span><span class="p">:</span><span class="n">d</span><span class="p">,</span> <span class="s">'j'</span><span class="p">:</span><span class="n">j</span><span class="p">})</span>
<span class="n">run</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="s">'127.0.0.1'</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="mi">8000</span><span class="p">,</span> <span class="n">debug</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</code></pre>
</div>
<p>唯一的一个 html 就是具体用 highcharts 画图的地方,如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nt"><html></span>
<span class="nt"><head></span>
<span class="nt"><meta</span> <span class="na">http-equiv=</span><span class="s">"refresh"</span> <span class="na">content=</span><span class="s">"60"</span><span class="nt">></span>
<span class="nt"><script </span><span class="na">type=</span><span class="s">"text/javascript"</span> <span class="na">src=</span><span class="s">"http://ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">type=</span><span class="s">"text/javascript"</span> <span class="na">src=</span><span class="s">"/js/highstock.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">type=</span><span class="s">"text/javascript"</span> <span class="na">src=</span><span class="s">"/js/highcharts.js"</span><span class="nt">></script></span>
<span class="nt"><script></span>
<span class="nx">$</span><span class="p">(</span><span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="nx">Highcharts</span><span class="p">.</span><span class="nx">setOptions</span><span class="p">({</span>
<span class="na">global</span><span class="p">:</span> <span class="p">{</span>
<span class="na">useUTC</span><span class="p">:</span> <span class="kc">false</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="nx">$</span><span class="p">.</span><span class="nx">getJSON</span><span class="p">(</span><span class="s1">'/boll/'</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">bolldata</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">ohlc</span> <span class="o">=</span> <span class="p">[]</span>
<span class="nx">volume</span> <span class="o">=</span> <span class="p">[],</span>
<span class="nx">dataLength</span> <span class="o">=</span> <span class="nx">bolldata</span><span class="p">[</span><span class="s1">'ohlc'</span><span class="p">].</span><span class="nx">length</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="nx">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="o"><</span> <span class="nx">dataLength</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">ohlc</span><span class="p">.</span><span class="nx">push</span><span class="p">([</span>
<span class="nx">bolldata</span><span class="p">[</span><span class="s1">'ohlc'</span><span class="p">][</span><span class="nx">i</span><span class="p">][</span><span class="mi">0</span><span class="p">],</span>
<span class="nx">bolldata</span><span class="p">[</span><span class="s1">'ohlc'</span><span class="p">][</span><span class="nx">i</span><span class="p">][</span><span class="mi">1</span><span class="p">],</span>
<span class="nx">bolldata</span><span class="p">[</span><span class="s1">'ohlc'</span><span class="p">][</span><span class="nx">i</span><span class="p">][</span><span class="mi">2</span><span class="p">],</span>
<span class="nx">bolldata</span><span class="p">[</span><span class="s1">'ohlc'</span><span class="p">][</span><span class="nx">i</span><span class="p">][</span><span class="mi">3</span><span class="p">],</span>
<span class="nx">bolldata</span><span class="p">[</span><span class="s1">'ohlc'</span><span class="p">][</span><span class="nx">i</span><span class="p">][</span><span class="mi">4</span><span class="p">],</span>
<span class="p">]);</span>
<span class="nx">volume</span><span class="p">.</span><span class="nx">push</span><span class="p">([</span>
<span class="nx">bolldata</span><span class="p">[</span><span class="s1">'ohlc'</span><span class="p">][</span><span class="nx">i</span><span class="p">][</span><span class="mi">0</span><span class="p">],</span>
<span class="nx">bolldata</span><span class="p">[</span><span class="s1">'ohlc'</span><span class="p">][</span><span class="nx">i</span><span class="p">][</span><span class="mi">5</span><span class="p">],</span>
<span class="p">])</span>
<span class="p">};</span>
<span class="nx">$</span><span class="p">.</span><span class="nx">getJSON</span><span class="p">(</span><span class="s1">'/kdj/'</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">kdjdata</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">.</span><span class="nx">getJSON</span><span class="p">(</span><span class="s1">'/macd/'</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">macddata</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">(</span><span class="s1">'#container'</span><span class="p">).</span><span class="nx">highcharts</span><span class="p">(</span><span class="s1">'StockChart'</span><span class="p">,</span> <span class="p">{</span>
<span class="na">rangeSelector</span><span class="p">:</span> <span class="p">{</span>
<span class="na">enabled</span><span class="p">:</span> <span class="mi">0</span>
<span class="p">},</span>
<span class="na">chart</span><span class="p">:</span> <span class="p">{</span>
<span class="na">backgroundColor</span><span class="p">:</span> <span class="s1">'#333333'</span><span class="p">,</span>
<span class="p">},</span>
<span class="na">tooltip</span><span class="p">:</span> <span class="p">{</span>
<span class="na">formatter</span><span class="p">:</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">s</span> <span class="o">=</span> <span class="s1">'<b>'</span><span class="o">+</span> <span class="nx">Highcharts</span><span class="p">.</span><span class="nx">dateFormat</span><span class="p">(</span><span class="s1">'%A, %b %e, %H:%M'</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="nx">x</span><span class="p">)</span> <span class="o">+</span><span class="s1">'</b>'</span><span class="p">;</span>
<span class="nx">$</span><span class="p">.</span><span class="nx">each</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">points</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">i</span><span class="p">,</span> <span class="nx">point</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">s</span> <span class="o">+=</span> <span class="s1">'<br/>'</span><span class="o">+</span><span class="k">this</span><span class="p">.</span><span class="nx">series</span><span class="p">.</span><span class="nx">name</span><span class="o">+</span><span class="s1">': '</span><span class="o">+</span><span class="nb">parseFloat</span><span class="p">(</span><span class="nx">point</span><span class="p">.</span><span class="nx">y</span><span class="p">).</span><span class="nx">toFixed</span><span class="p">(</span><span class="mi">2</span><span class="p">);</span>
<span class="p">});</span>
<span class="k">return</span> <span class="nx">s</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="na">plotOptions</span><span class="p">:</span> <span class="p">{</span>
<span class="na">series</span><span class="p">:</span> <span class="p">{</span>
<span class="na">marker</span><span class="p">:</span> <span class="p">{</span>
<span class="na">enabled</span><span class="p">:</span> <span class="kc">false</span>
<span class="p">},</span>
<span class="na">lineWidth</span><span class="p">:</span> <span class="mf">1.1</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="na">yAxis</span><span class="p">:</span> <span class="p">[{</span>
<span class="na">title</span><span class="p">:</span> <span class="p">{</span>
<span class="na">text</span><span class="p">:</span> <span class="s1">'MACD(12,26,9)'</span>
<span class="p">},</span>
<span class="na">height</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
<span class="p">},</span> <span class="p">{</span>
<span class="na">title</span><span class="p">:</span> <span class="p">{</span>
<span class="na">text</span><span class="p">:</span> <span class="s1">'KDJ(9,3,3)'</span>
<span class="p">},</span>
<span class="na">top</span><span class="p">:</span> <span class="mi">250</span><span class="p">,</span>
<span class="na">height</span><span class="p">:</span> <span class="mi">150</span><span class="p">,</span>
<span class="na">offset</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
<span class="na">gridLineDashStyle</span><span class="p">:</span> <span class="s1">'Dash'</span><span class="p">,</span>
<span class="na">tickPositions</span><span class="p">:</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">50</span><span class="p">,</span> <span class="mi">80</span><span class="p">,</span> <span class="mi">100</span><span class="p">,</span> <span class="mi">200</span><span class="p">]</span>
<span class="p">},</span> <span class="p">{</span>
<span class="na">title</span><span class="p">:</span> <span class="p">{</span>
<span class="na">text</span><span class="p">:</span> <span class="s1">'BOLL(20)'</span>
<span class="p">},</span>
<span class="na">top</span><span class="p">:</span> <span class="mi">450</span><span class="p">,</span>
<span class="na">height</span><span class="p">:</span> <span class="mi">300</span><span class="p">,</span>
<span class="na">offset</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
<span class="p">},</span> <span class="p">{</span>
<span class="na">title</span><span class="p">:</span> <span class="p">{</span>
<span class="na">text</span><span class="p">:</span> <span class="s1">'VOL'</span>
<span class="p">},</span>
<span class="na">top</span><span class="p">:</span> <span class="mi">800</span><span class="p">,</span>
<span class="na">height</span><span class="p">:</span> <span class="mi">100</span><span class="p">,</span>
<span class="na">offset</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
<span class="p">}],</span>
<span class="na">series</span><span class="p">:</span> <span class="p">[{</span>
<span class="na">name</span><span class="p">:</span> <span class="s1">'BAR'</span><span class="p">,</span>
<span class="na">color</span><span class="p">:</span> <span class="s1">''</span><span class="p">,</span>
<span class="na">negativeColor</span><span class="p">:</span> <span class="s1">''</span><span class="p">,</span>
<span class="na">borderColor</span><span class="p">:</span> <span class="s1">'#333333'</span><span class="p">,</span>
<span class="na">type</span><span class="p">:</span> <span class="s1">'column'</span><span class="p">,</span>
<span class="na">data</span><span class="p">:</span> <span class="nx">macddata</span><span class="p">[</span><span class="s1">'bar'</span><span class="p">],</span>
<span class="na">yAxis</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
<span class="p">},</span> <span class="p">{</span>
<span class="na">name</span><span class="p">:</span> <span class="s1">'DIFF'</span><span class="p">,</span>
<span class="na">color</span><span class="p">:</span> <span class="s1">'#ffffff'</span><span class="p">,</span>
<span class="na">type</span><span class="p">:</span> <span class="s1">'line'</span><span class="p">,</span>
<span class="na">data</span><span class="p">:</span> <span class="nx">macddata</span><span class="p">[</span><span class="s1">'dif'</span><span class="p">],</span>
<span class="na">lineWidth</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
<span class="na">yAxis</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
<span class="p">},</span> <span class="p">{</span>
<span class="na">name</span><span class="p">:</span> <span class="s1">'DEA'</span><span class="p">,</span>
<span class="na">color</span><span class="p">:</span> <span class="s1">'#ffff00'</span><span class="p">,</span>
<span class="na">type</span><span class="p">:</span> <span class="s1">'line'</span><span class="p">,</span>
<span class="na">data</span><span class="p">:</span> <span class="nx">macddata</span><span class="p">[</span><span class="s1">'dea'</span><span class="p">],</span>
<span class="na">lineWidth</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
<span class="na">yAxis</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
<span class="p">},</span> <span class="p">{</span>
<span class="na">name</span><span class="p">:</span> <span class="s1">'K'</span><span class="p">,</span>
<span class="na">color</span><span class="p">:</span> <span class="s1">'#ffffff'</span><span class="p">,</span>
<span class="na">type</span><span class="p">:</span> <span class="s1">'line'</span><span class="p">,</span>
<span class="na">data</span><span class="p">:</span> <span class="nx">kdjdata</span><span class="p">[</span><span class="s1">'k'</span><span class="p">],</span>
<span class="na">yAxis</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="p">},</span> <span class="p">{</span>
<span class="na">name</span><span class="p">:</span> <span class="s1">'D'</span><span class="p">,</span>
<span class="na">color</span><span class="p">:</span> <span class="s1">'#ffff00'</span><span class="p">,</span>
<span class="na">type</span><span class="p">:</span> <span class="s1">'line'</span><span class="p">,</span>
<span class="na">data</span><span class="p">:</span> <span class="nx">kdjdata</span><span class="p">[</span><span class="s1">'d'</span><span class="p">],</span>
<span class="na">yAxis</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="p">},</span> <span class="p">{</span>
<span class="na">name</span><span class="p">:</span> <span class="s1">'J'</span><span class="p">,</span>
<span class="na">color</span><span class="p">:</span> <span class="s1">'#cc99cc'</span><span class="p">,</span>
<span class="na">type</span><span class="p">:</span> <span class="s1">'line'</span><span class="p">,</span>
<span class="na">data</span><span class="p">:</span> <span class="nx">kdjdata</span><span class="p">[</span><span class="s1">'j'</span><span class="p">],</span>
<span class="na">yAxis</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="p">},</span> <span class="p">{</span>
<span class="na">type</span><span class="p">:</span> <span class="s1">'candlestick'</span><span class="p">,</span>
<span class="na">name</span><span class="p">:</span> <span class="s1">'ohlc'</span><span class="p">,</span>
<span class="na">data</span><span class="p">:</span> <span class="nx">ohlc</span><span class="p">,</span>
<span class="na">upColor</span><span class="p">:</span> <span class="s1">''</span><span class="p">,</span>
<span class="na">upLineColor</span><span class="p">:</span> <span class="s1">''</span><span class="p">,</span>
<span class="na">color</span><span class="p">:</span> <span class="s1">''</span><span class="p">,</span>
<span class="na">lineColor</span><span class="p">:</span> <span class="s1">''</span><span class="p">,</span>
<span class="na">yAxis</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
<span class="p">},</span> <span class="p">{</span>
<span class="na">type</span><span class="p">:</span> <span class="s1">'spline'</span><span class="p">,</span>
<span class="na">name</span><span class="p">:</span> <span class="s1">'up'</span><span class="p">,</span>
<span class="na">data</span><span class="p">:</span> <span class="nx">bolldata</span><span class="p">[</span><span class="s1">'up'</span><span class="p">],</span>
<span class="na">color</span><span class="p">:</span> <span class="s1">'#ffff00'</span><span class="p">,</span>
<span class="na">lineWidth</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
<span class="na">yAxis</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
<span class="p">},</span> <span class="p">{</span>
<span class="na">type</span><span class="p">:</span> <span class="s1">'spline'</span><span class="p">,</span>
<span class="na">name</span><span class="p">:</span> <span class="s1">'md'</span><span class="p">,</span>
<span class="na">data</span><span class="p">:</span> <span class="nx">bolldata</span><span class="p">[</span><span class="s1">'md'</span><span class="p">],</span>
<span class="na">color</span><span class="p">:</span> <span class="s1">'#ffffff'</span><span class="p">,</span>
<span class="na">lineWidth</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
<span class="na">yAxis</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
<span class="p">},</span> <span class="p">{</span>
<span class="na">type</span><span class="p">:</span> <span class="s1">'spline'</span><span class="p">,</span>
<span class="na">name</span><span class="p">:</span> <span class="s1">'dn'</span><span class="p">,</span>
<span class="na">data</span><span class="p">:</span> <span class="nx">bolldata</span><span class="p">[</span><span class="s1">'dn'</span><span class="p">],</span>
<span class="na">color</span><span class="p">:</span> <span class="s1">'#cc99cc'</span><span class="p">,</span>
<span class="na">lineWidth</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
<span class="na">yAxis</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
<span class="p">},</span> <span class="p">{</span>
<span class="na">name</span><span class="p">:</span> <span class="s1">'VOL'</span><span class="p">,</span>
<span class="na">borderColor</span><span class="p">:</span> <span class="s1">'#333333'</span><span class="p">,</span>
<span class="na">type</span><span class="p">:</span> <span class="s1">'column'</span><span class="p">,</span>
<span class="na">data</span><span class="p">:</span> <span class="nx">volume</span><span class="p">,</span>
<span class="na">yAxis</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span>
<span class="p">}]</span>
<span class="p">});</span>
<span class="p">});</span>
<span class="p">});</span>
<span class="p">});</span>
<span class="p">});</span>
<span class="nt"></script></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><div</span> <span class="na">id=</span><span class="s">"container"</span> <span class="na">style=</span><span class="s">"min-width:800px;height:1000px;"</span><span class="nt">></div></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</code></pre>
</div>
<p>highcharts 有个问题,就是不能跟 amcharts 或者 echarts 那样提供一个画笔工具,让用户自己在生成的图形上再涂抹线条,这个功能其实在蜡烛图上判断压力位支撑位的时候很有用。不过蜡烛图 btc123 也提供了,我也就懒得再用 amcharts 重写一遍。</p>
<p>效果如下:</p>
<p><img src="/images/uploads//btc.png" alt="" /></p>
为 gitolite 实现 mailinglist 命令行操控
2013-12-09T00:00:00+08:00
perl
git
linux
ssh
http://chenlinux.com/2013/12/09/add-mailinglist-command-to-gitolite
<p>gitolite 是一个很常用的 git 仓库管理软件,可以通过命令行方式便捷操作自己拥有权限的项目仓库。不过不是所有的操作都可以通过命令完成,很多还是需要通知 gitolite 管理员来统一修改配置然后生效。比如通过 hook 发邮件这件事情。邮件收件人地址肯定每个项目就不一样,这个还要让管理员逐一来改,就不太好。所以这里实现了一个 mailinglist 的命令行操作子命令。</p>
<h1 id="section">使用说明</h1>
<ol>
<li>要求.gitolite.rc 中开启 GIT_CONFIG_KEYS 允许 hooks</li>
<li>要求.gitolite.rc 中开启 ENABLE 允许 mailinglist</li>
<li>在.gitolite/hooks/common/ 下软连接 git 默认的 post-receive-email 成 post-receive 文件</li>
</ol>
<p>注意这里修改的 hooks 是针对整个 gitolite 的总 hooks 目录。而不是每个 repo 自己的 hooks,这个是单独有 repo-special-hooks 命令来管理的。</p>
<h1 id="section-1">代码修改</h1>
<p>代码上的修改主要就是两处:</p>
<ul>
<li>新增文件 src/commands/mailinglist 如下;</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="c1">#!/usr/bin/perl</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Data::</span><span class="nv">Dumper</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">lib</span> <span class="nv">$ENV</span><span class="p">{</span><span class="nv">GL_LIBDIR</span><span class="p">};</span>
<span class="k">use</span> <span class="nn">Gitolite::</span><span class="nv">Rc</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Gitolite::</span><span class="nv">Common</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Gitolite::</span><span class="nv">Easy</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Gitolite::Conf::</span><span class="nv">Load</span><span class="p">;</span>
<span class="k">our</span> <span class="nv">%one_repo</span><span class="p">;</span>
<span class="k">our</span> <span class="nv">%one_config</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$repo</span> <span class="o">=</span> <span class="nv">$ARGV</span><span class="p">[</span><span class="mi">0</span><span class="p">];</span>
<span class="k">my</span> <span class="nv">$addr</span> <span class="o">=</span> <span class="nv">$ARGV</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span>
<span class="nv">$ENV</span><span class="p">{</span><span class="nv">GL_USER</span><span class="p">}</span> <span class="ow">or</span> <span class="nv">_die</span> <span class="s">"GL_USER not set"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$generic_error</span> <span class="o">=</span> <span class="s">"repo does not exist, or you are not authorised"</span><span class="p">;</span>
<span class="nv">_die</span> <span class="nv">$generic_error</span> <span class="k">if</span> <span class="ow">not</span> <span class="nv">owns</span><span class="p">(</span><span class="nv">$repo</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$addr</span> <span class="p">)</span> <span class="p">{</span>
<span class="c1"># write</span>
<span class="nv">_chdir</span><span class="p">(</span><span class="s">"$rc{GL_REPO_BASE}/$repo.git"</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span> <span class="o">-</span><span class="nv">f</span> <span class="s">"gl-conf"</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$cc</span> <span class="o">=</span> <span class="s">"./gl-conf"</span><span class="p">;</span>
<span class="nv">_die</span> <span class="s">"parse '$cc' failed: "</span> <span class="o">.</span> <span class="p">(</span> <span class="vg">$!</span> <span class="ow">or</span> <span class="vg">$@</span> <span class="p">)</span> <span class="k">unless</span> <span class="k">do</span> <span class="nv">$cc</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">my</span> <span class="nv">$num</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$i</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span> <span class="nv">@</span><span class="p">{</span> <span class="nv">$one_config</span><span class="p">{</span><span class="nv">$repo</span><span class="p">}</span> <span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$num</span> <span class="o">=</span> <span class="nv">$_</span><span class="o">-></span><span class="p">[</span><span class="mi">0</span><span class="p">];</span>
<span class="k">next</span> <span class="k">unless</span> <span class="nv">$_</span><span class="o">-></span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="ow">eq</span> <span class="s">'hooks.mailinglist'</span><span class="p">;</span>
<span class="nv">$_</span><span class="o">-></span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="nv">$addr</span> <span class="ow">and</span> <span class="nv">$i</span><span class="o">++</span> <span class="ow">and</span> <span class="k">last</span><span class="p">;</span>
<span class="p">}</span>
<span class="nb">push</span> <span class="nv">@</span><span class="p">{</span><span class="nv">$one_config</span><span class="p">{</span><span class="nv">$repo</span><span class="p">}},</span> <span class="p">[</span> <span class="nv">$num</span><span class="p">,</span> <span class="s">'hooks.mailinglist'</span><span class="p">,</span> <span class="nv">$addr</span><span class="p">]</span> <span class="k">unless</span> <span class="nv">$i</span><span class="p">;</span>
<span class="nb">open</span><span class="p">(</span> <span class="k">my</span> <span class="nv">$compiled_fh</span><span class="p">,</span> <span class="s">">"</span><span class="p">,</span> <span class="s">"gl-conf"</span> <span class="p">)</span> <span class="ow">or</span> <span class="nv">_die</span> <span class="vg">$!</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$dumped_data</span> <span class="o">=</span> <span class="s">''</span><span class="p">;</span>
<span class="nv">$dumped_data</span> <span class="o">=</span> <span class="nn">Data::</span><span class="nv">Dumper</span><span class="o">-></span><span class="nv">Dump</span><span class="p">(</span> <span class="p">[</span> <span class="o">\</span><span class="nv">%one_repo</span> <span class="p">],</span> <span class="p">[</span><span class="sx">qw(*one_repo)</span><span class="p">]</span> <span class="p">);</span>
<span class="nv">$dumped_data</span> <span class="o">.=</span> <span class="nn">Data::</span><span class="nv">Dumper</span><span class="o">-></span><span class="nv">Dump</span><span class="p">(</span> <span class="p">[</span> <span class="o">\</span><span class="nv">%one_config</span> <span class="p">],</span> <span class="p">[</span><span class="sx">qw(*one_config)</span><span class="p">]</span> <span class="p">);</span>
<span class="k">print</span> <span class="nv">$compiled_fh</span> <span class="nv">$dumped_data</span><span class="p">;</span>
<span class="nb">close</span> <span class="nv">$compiled_fh</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="c1"># read</span>
<span class="k">my</span> <span class="nv">$val</span> <span class="o">=</span> <span class="nv">git_config</span><span class="p">(</span><span class="nv">$repo</span><span class="p">,</span> <span class="s">'hooks.mailinglist'</span><span class="p">);</span>
<span class="k">print</span> <span class="nv">$val</span><span class="o">-></span><span class="p">{</span><span class="s">'hooks.mailinglist'</span><span class="p">};</span>
<span class="p">}</span>
<span class="mi">1</span><span class="p">;</span>
</code></pre>
</div>
<ul>
<li>
<p>修改 src/lib/Gitolite/Rc.pm 如下:</p>
<p>@@ -459,7 +459,7 @@ <strong>DATA</strong><br />
UMASK => 0077,</p>
<div class="highlighter-rouge"><pre class="highlight"><code> # look for "git-config" in the documentation - GIT_CONFIG_KEYS => '', + GIT_CONFIG_KEYS => 'hooks.*',
# comment out if you don't need all the extra detail in the logfile
LOG_EXTRA => 1, @@ -520,6 +520,7 @@ __DATA__
'info',
'perms',
'writable', + 'mailinglist',
</code></pre>
</div>
</li>
</ul>
<p>gitolite 本身相关的代码解析,和实现思路,我写成了这个 slide,欢迎观看:</p>
<div><embed src='http://www.docin.com/DocinViewer-737880351-144.swf' width='100%' height='600' type=application/x-shockwave-flash ALLOWFULLSCREEN='true' ALLOWSCRIPTACCESS='always'></div>
Puppet 的类参数传递
2013-11-04T00:00:00+08:00
puppet
http://chenlinux.com/2013/11/04/puppet-class-parameter
<p>之前使用 ENC 管理 puppet,尽量保持了输出 yaml 内容的简单,只提供了一个统一的全局参数定义 node 的 role。(题外话,puppetlabs 推荐了另一个通过继承关系实现 role 的示例,见:<a href="http://www.craigdunn.org/2012/05/239/">Designing Puppet - Roles and Profiles</a>。)</p>
<p>但是 puppet 中有些配置确实修改比较频繁,文件操作不得不说是一件不甚方便的事情,于是重新考虑通过类参数的方式来灵活化某些配置的操作。</p>
<h1 id="section">修改前</h1>
<h3 id="nginxmanifestsinitpp">nginx/manifests/init.pp</h3>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="n">nginx</span> <span class="p">{</span>
<span class="kp">include</span> <span class="s2">"nginx::${::role}"</span>
<span class="p">}</span>
</code></pre>
</div>
<h3 id="nginxmanifestsloadbalancerpp">nginx/manifests/loadbalancer.pp</h3>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="n">nginx</span><span class="o">::</span><span class="n">loadbalancer</span> <span class="p">{</span>
<span class="vg">$iplist</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'192.168.0.2:80'</span><span class="p">]</span>
<span class="n">file</span> <span class="p">{</span> <span class="s1">'nginx.conf'</span><span class="p">:</span>
<span class="n">content</span> <span class="o">=></span> <span class="n">template</span><span class="p">(</span><span class="s1">'nginx/nginx.conf.erb'</span><span class="p">),</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<h3 id="enc-nginxhostname">enc nginxhostname</h3>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nn">---</span>
<span class="s">classes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">nginx</span>
<span class="pi">-</span> <span class="s">base</span>
<span class="s">environment</span><span class="pi">:</span> <span class="s">production</span>
<span class="s">parameters</span><span class="pi">:</span>
<span class="s">role</span><span class="pi">:</span> <span class="s">loadbalancer</span>
</code></pre>
</div>
<h1 id="section-1">修改后</h1>
<h3 id="nginxmanifestsinitpp-1">nginx/manifests/init.pp</h3>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="n">nginx</span> <span class="p">(</span><span class="vg">$iplist</span> <span class="o">=</span> <span class="p">[])</span> <span class="p">{</span>
<span class="k">class</span> <span class="p">{</span> <span class="s2">"nginx::${::role}"</span><span class="p">:</span>
<span class="n">iplist</span> <span class="o">=></span> <span class="vg">$iplist</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<h3 id="nginxmanifestsloadbalancerpp-1">nginx/manifests/loadbalancer.pp</h3>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="n">nginx</span><span class="o">::</span><span class="n">loadbalancer</span> <span class="p">(</span><span class="vg">$iplist</span> <span class="o">=</span> <span class="p">[])</span> <span class="p">{</span>
<span class="n">file</span> <span class="p">{</span> <span class="s1">'nginx.conf'</span><span class="p">:</span>
<span class="n">content</span> <span class="o">=></span> <span class="n">template</span><span class="p">(</span><span class="s1">'nginx/nginx.conf.erb'</span><span class="p">),</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<h3 id="enc-nginxhostname-1">enc nginxhostname</h3>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nn">---</span>
<span class="s">classes</span><span class="pi">:</span>
<span class="s">nginx</span><span class="pi">:</span>
<span class="s">iplist</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">192.168.0.2:80</span>
<span class="s">base</span><span class="pi">:</span> <span class="s">~</span>
<span class="s">environment</span><span class="pi">:</span> <span class="s">production</span>
<span class="s">parameters</span><span class="pi">:</span>
<span class="s">role</span><span class="pi">:</span> <span class="s">loadbalancer</span>
</code></pre>
</div>
<h1 id="section-2">要点</h1>
<ol>
<li>虽然真正需要 $iplist 的是下面的一个子类,但是 ENC 传值是给的父类,所以需要一层层传递下去;</li>
<li>ENC 中给类传参,类就要写成哈希形式,否则是数组形式;</li>
<li>有参数的类,在调用的时候无法使用 <code class="highlighter-rouge">include</code> 形式的写法,只能用资源调用形式的写法。</li>
</ol>
<p>修改中出现了一个很搞笑的错误,因为是在 vim 里批量转换,结果子类名字后面多了一个空格,成了<code class="highlighter-rouge">class { "nginx::${::role} ":</code>这样。结果 puppet 一直返回报错说 “Invalid Parameter”。这时候一个习惯性的思维造成了误会:我们一般会认为<code class="highlighter-rouge">:</code>后面的那一行行键值对才是 parameter,但其实这里子类名也是 <code class="highlighter-rouge">class</code> 这个资源调用的 parameter。当然,如果可以在这里报一个 <code class="highlighter-rouge">Class Not Found</code> 就更好了。</p>
用 Perl 读取通达信日线数据
2013-11-04T00:00:00+08:00
perl
http://chenlinux.com/2013/11/04/perl-unpack-tongdaxin
<p>之前看 skyline 的报警机制的时候,为了寻找测试数据,曾经想到是不是可以用股价走势。其实股价走势分析也是一个很深的编程领域,有些选股软件一份就好几千的卖。当然我这里没兴趣和时间搞那么复杂了。简单的说一下如何从通达信的存档里读取日线数据,说到底还是 <code class="highlighter-rouge">pack/unpack</code> 的运用:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!perl</span>
<span class="nb">open</span> <span class="k">my</span> <span class="nv">$fh</span><span class="p">,</span> <span class="s">'<'</span><span class="p">,</span> <span class="s">'C:\new_sxzq_v6\vipdoc\sh\lday\sh000001.day'</span><span class="p">;</span>
<span class="k">while</span> <span class="p">(</span> <span class="nb">sysread</span> <span class="nv">$fh</span><span class="p">,</span> <span class="k">my</span> <span class="nv">$buf</span><span class="p">,</span> <span class="mi">32</span> <span class="p">)</span> <span class="p">{</span>
<span class="c1"># 日期,开盘,最高,最低,收盘,成交金额,成交量,预留位</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$date</span><span class="p">,</span> <span class="nv">$open</span><span class="p">,</span> <span class="nv">$high</span><span class="p">,</span> <span class="nv">$low</span><span class="p">,</span> <span class="nv">$close</span><span class="p">,</span> <span class="nv">$amount</span><span class="p">,</span> <span class="nv">$vol</span><span class="p">,</span> <span class="nv">$reserved</span> <span class="p">)</span> <span class="o">=</span>
<span class="nb">unpack</span><span class="p">(</span> <span class="s">'Ii4fi2'</span><span class="p">,</span> <span class="nv">$buf</span> <span class="p">);</span>
<span class="nb">printf</span> <span class="s">"%s %.2f %.2f %.2f %.2f %d %d\n"</span><span class="p">,</span> <span class="nv">$date</span><span class="p">,</span> <span class="nv">$open</span> <span class="sr">/ 100, $high /</span> <span class="mi">100</span><span class="p">,</span>
<span class="nv">$low</span> <span class="sr">/ 100, $close /</span> <span class="mi">100</span><span class="p">,</span> <span class="nv">$amount</span><span class="p">,</span> <span class="nv">$vol</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
<p>注意这里一定要一边 <code class="highlighter-rouge">sysread</code> 一边 <code class="highlighter-rouge">while</code>,否则一只股票的历史(上例中是上证指数)都没读完就会内存溢出的。</p>
selinux 对 webserver 文件发布的影响
2013-10-26T00:00:00+08:00
linux
apache
selinux
http://chenlinux.com/2013/10/26/selinux-affect-to-apache-documentroot
<p>SELinux 在国内是一个很少有人用的东西,一般来说,服务器上手第一件事情就是把 SELinux 关掉,以至于有问题的时候排查思路里都压根没检查 SELinux 这步。</p>
<p>昨天在个人电脑的 Fedora 上搭建一个 webserver 发布几个文件,本来想着简单任务越快越好,几行命令完成:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sudo yum install httpd
sudo mv ~/src /var/www/html/
sudo service httpd start
</code></pre>
</div>
<p>结果居然一直返回 <code class="highlighter-rouge">403 Denied</code>!</p>
<p>看 httpd 的 error.log ,一直报这么一行错误:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[core:error] [pid 3806] (13)Permission denied: [client 127.0.0.1:59180] AH00035: access to /src/master.zip denied (filesystem path '/var/www/html/src') because search permissions are missing on a component of the path
</code></pre>
</div>
<p>很奇怪吧,于是我先去确认了 httpd.conf 里关于 <code class="highlighter-rouge"><Directory "/var/www/html"></code> 的配置(因为 Fedora19 的 httpd 版本是 2.4.6,我以为新版本有变化了),然后去确认了 <code class="highlighter-rouge">/var/www/html/src</code> 的权限是不是 755,其他用户可读的。都没问题!</p>
<p>最后还是在 apache 的 httpd 官方文档上找到了关于这个错误码的详细解释,原来还有一种可能性,就是 SELinux 的安全控制!这个可以通过下面这个命令看到:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="gp">$ </span>ls -lZ /var/www /var/www/html
/var/www:
drwxr-xr-x. root root system_u:object_r:httpd_sys_script_exec_t:s0 cgi-bin
drwxr-xr-x. apache apache system_u:object_r:httpd_sys_content_t:s0 html
/var/www/html:
drwxr-xr-x. chenlin.rao chenlin.rao unconfined_u:object_r:user_home_t:s0 src
</code></pre>
</div>
<p>看到没有,这里这些文件的 SELinux 类型是不一样的,默认的 <code class="highlighter-rouge">/var/www/html</code> 是 <code class="highlighter-rouge">httpd_sys_content_t</code>,<code class="highlighter-rouge">/var/www/cgi-bin</code> 是 <code class="highlighter-rouge">httpd_sys_script_exec_t</code>,而从我家目录移过去的 <code class="highlighter-rouge">/var/www/html/src</code> 是 <code class="highlighter-rouge">user_home_t</code>!</p>
<p>解决办法也很简单,把这个类型也改过来就好了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="gp">$ </span>chcon -R -t httpd_sys_content_t /var/www/html/src
</code></pre>
</div>
<p>这是第一次接触 SELinux 的安全管理,真的是好细致!</p>
用 plenv 代替 perlbrew 管理 Perl5
2013-10-25T00:00:00+08:00
perl
bash
ruby
http://chenlinux.com/2013/10/25/intro-plenv
<p>我们都知道有 virtualenv 啊,rvm 啊之类的工具来管理 python,ruby的多版本问题,后来台湾的朋友也引入到了 Perl 世界,这就是 perlbrew。</p>
<p>不过 perlbrew 在使用的时候,有个非常让我不理解的地方,就是切换 Perl 版本后,整个终端的环境变量都被清空了。后来发现了一个新项目,叫 plenv,没错,一眼就可以看出来这是 rlenv 工具的 Perl 版。</p>
<p>和 perlbrew 不一样,目前版本的 plenv 已经是一个纯粹的 shell 工具。说起来原先一直是 shell 工具的 rvm,这周却在募捐准备改用 Ruby 重写了(据说是因为已经 2 万行的 bash 代码,作者快控制不住了)。</p>
<p>用法非常简单:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>git clone git://github.com/tokuhirom/plenv.git ~/.plenv
<span class="nb">echo</span> <span class="s1">'export PATH="$HOME/.plenv/bin:$PATH"'</span> >> ~/.bash_profile
<span class="nb">echo</span> <span class="s1">'eval "$(plenv init -)"'</span> >> ~/.bash_profile
<span class="nb">exec</span> <span class="nv">$SHELL</span> -l <span class="c"># 这步相当于退出重登录终端</span>
git clone git://github.com/tokuhirom/Perl-Build.git ~/.plenv/plugins/perl-build/
plenv install 5.18.0
plenv rehash <span class="c"># 每次在 $HOME/.plenv/bin 下安装了新的命令后都要执行一次这个</span>
plenv install-cpanm
plenv rehash
plenv shell 5.18.0 <span class="c"># 还有 global 和 local 两者可设</span>
</code></pre>
</div>
<p>目前我已经用 plenv 管理自己电脑上的 Perl5 了,你们呢?</p>
Perl 的 overload 妙用
2013-10-16T00:00:00+08:00
perl
mojolicious
OOP
http://chenlinux.com/2013/10/16/perl-overload
<p>在使用 Mojolicious 的时候,通常我们会发现一个很有趣的现象。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nv">ojo</span><span class="p">;</span>
<span class="nv">say</span> <span class="nv">g</span><span class="p">(</span><span class="s">'http://www.baidu.com'</span><span class="p">)</span><span class="o">-></span><span class="nv">dom</span><span class="o">-></span><span class="nv">at</span><span class="p">(</span><span class="s">'script'</span><span class="p">);</span>
<span class="nv">say</span> <span class="nv">g</span><span class="p">(</span><span class="s">'http://www.baidu.com'</span><span class="p">)</span><span class="o">-></span><span class="nv">dom</span><span class="o">-></span><span class="nv">at</span><span class="p">(</span><span class="s">'script'</span><span class="p">)</span><span class="o">-></span><span class="nv">text</span><span class="p">;</span>
</code></pre>
</div>
<p>这里可以看到,在用 <code class="highlighter-rouge">at()</code> 方法之后得到的结果,如果从上一行解读,似乎应该是一个字符串;但是从下一行解读,又还是一个对象,可以继续调用 <code class="highlighter-rouge">->text</code> 属性。</p>
<p>Perl 本身不是一个纯对象式的语言,字符串本身是没有对象属性的。而直接打印对象的话,应该输出的是类似 <code class="highlighter-rouge">Mojo::DOM->HASH(0x1234567)</code> 的效果。那这个效果是怎么实现的呢?</p>
<p>翻了 Mojo 的代码之后,发现原来 Mojo 里是把字符串、数组等都实现成了对象,分别是 <code class="highlighter-rouge">Mojo::ByteStream</code> 和 <code class="highlighter-rouge">Mojo::Collection</code> 两个类。然后再实现中,运用了 <code class="highlighter-rouge">overload</code> 来实现这个效果。代码很简单,<code class="highlighter-rouge">Mojo::ByteStream</code> 里是这样的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nv">overload</span> <span class="s">'""'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="nb">shift</span><span class="o">-></span><span class="nv">to_string</span> <span class="p">},</span> <span class="nv">fallback</span> <span class="o">=></span> <span class="mi">1</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">to_string</span> <span class="p">{</span> <span class="nv">$</span><span class="p">{</span><span class="nv">$_</span><span class="p">[</span><span class="mi">0</span><span class="p">]}</span> <span class="p">}</span>
</code></pre>
</div>
<p>此外, <code class="highlighter-rouge">Mojo::DOM</code>,<code class="highlighter-rouge">Mojo::URL</code>,<code class="highlighter-rouge">Mojo::JSON</code> 等十多个类中都用了这个方法。</p>
<p>看起来似乎还不是很明了,再贴两段 <a href="http://perldoc.perl.org/overload.html">overload 的 POD</a> 就清楚了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>It also defines an anonymous subroutine to implement stringification: this is called whenever an object blessed into the package Number is used in a string context...
For example, the subroutine for '""' (stringify) may be used where the overloaded object is passed as an argument to print,...
</code></pre>
</div>
<p>这下清楚了吧。一旦在某个类里 <code class="highlighter-rouge">overload</code> 了双引号,那么这个类的对象在标量环境下调用的时候就会先调用这个函数。最典型的例子就是用在 <code class="highlighter-rouge">print</code> 的时候。</p>
<p>下面我们可以自己也试试:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nb">package</span> <span class="nv">Test</span> <span class="mf">0.01</span> <span class="p">{</span>
<span class="k">use</span> <span class="nv">overload</span> <span class="s">'""'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="nb">join</span> <span class="s">" overloaded.\n"</span><span class="p">,</span> <span class="nv">@</span><span class="p">{</span><span class="o">+</span><span class="nb">shift</span><span class="p">}</span> <span class="p">};</span>
<span class="k">sub </span><span class="nf">new</span> <span class="p">{</span> <span class="nb">bless</span> <span class="p">[</span><span class="nv">@_</span><span class="p">[</span><span class="mi">1</span> <span class="o">..</span> <span class="nv">$#_</span><span class="p">]],</span> <span class="nb">shift</span> <span class="p">};</span>
<span class="p">}</span>
<span class="k">my</span> <span class="nv">$obj</span> <span class="o">=</span> <span class="k">new</span> <span class="nv">Test</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">2</span><span class="p">);</span>
<span class="k">print</span> <span class="nv">$obj</span><span class="p">;</span>
</code></pre>
</div>
<p>输出结果:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1 overloaded.
3 overloaded.
2
</code></pre>
</div>
【翻译】用 elasticsearch 和 logstash 为数十亿次客户搜索提供服务
2013-10-09T00:00:00+08:00
logstash
elasticsearch
http://chenlinux.com/2013/10/09/using-elasticsearch-and-logstash-to-serve-billions-of-searchable-events-for-customers
<p>原文地址:<a href="http://www.elasticsearch.org/blog/using-elasticsearch-and-logstash-to-serve-billions-of-searchable-events-for-customers/">http://www.elasticsearch.org/blog/using-elasticsearch-and-logstash-to-serve-billions-of-searchable-events-for-customers/</a></p>
<hr />
<p><em>今天非常高兴的欢迎我们的第一个外来博主,Rackspace软件开发工程师,目前为Mailgun工作的 <a href="https://twitter.com/ralphm">Ralph Meijer</a>。我们在 <a href="http://monitorama.eu/">Monitorama EU</a> 会面后,Ralph 提出可以给我们写一篇 Mailgun 里如何使用 Elasticsearch 的文章。他本人也早就活跃在 Elasticsearch 社区,经常参加我们在荷兰的聚会了。</em></p>
<p><img src="http://www.elasticsearch.org/content/uploads/2013/09/mailgun_150-300x85.png" alt="" /></p>
<p><a href="http://www.mailgun.com/">Mailgun</a> 收发大量电子邮件,我们跟踪和存储每封邮件发生的每个事件。每个月会新增数十亿事件,我们都要展示给我们的客户,方便他们很容易的分析数据,也就是全文搜索。下文是我们利用Elasticsearch和Logstash技术完成这个需求的技术细节(很高兴刚写完这篇文章就听说《<a href="http://elasticsearch.com/blog/welcome-jordan-logstash/">Logstash加入Elasticsearch</a>》了)。</p>
<h1 id="section">事件</h1>
<p>在 Mailgun 里,event可能是如下几种:进来一条信息,可能被接收可能被拒绝;出去一条信息,可能被投递可能被拒绝(垃圾信息或者反弹);信息是直接打开还是通过链接点击打开;收件人要求退订。所有这些事件,都有一些元信息可以帮助我们客户找出他们的信息什么时候,为什么,发生了什么。这个元信息包括:信息的发送者,收件人地址,信息id,SMTP错误码,链接URL,geo地理位置等等。</p>
<p>每个事件都是由一个时间戳和一系列字段构成的。一个典型的事件就是一个关联数组,或者叫字典、哈希表。</p>
<h1 id="section-1">事件访问设计</h1>
<p>假设我们已经有了各种事件,现在需要一个办法来给客户使用。在Mailgun的控制面板里,有一个日志标签,可以以时间倒序展示事件日志,并且还可以通过域名和级别来过滤日志,示例如下:</p>
<p><img src="http://9791b61a81187466cf77-03e2fb40b56101ddc8886446c68cb0c1.r77.cf2.rackcdn.com/mailgun_log_sample.png" alt="" /></p>
<p>在这个示例里,这个事件的级别是”warn”,因为SMTP错误码说明这是一个临时性问题,我们稍后会重试投递。这里有两个字段,一个时间戳,一个还没格式化的非结构化文本信息。为了醒目,这里我们会根据级别的不同给事件上不同的底色。</p>
<p>在这个网页之后,我还有一个接收日志的API,一个设置触发报警的hook页面。后面的报警完全是结构化了的带有很多元数据字段的JSON文档。比如,SMTP错误码有自己的字段,收件人地址和邮件标题等也都有。</p>
<p>不幸的是,原有的日志API非常有限。他只能返回邮件投递时间和控制面板里展示的非结构化的文本内容。没办法获取或者搜索多个字段(像报警页面里那样),更不要说全文搜索了。简单说,就是控制面板缺乏全文搜索。</p>
<h1 id="elasticsearch">用elasticsearch存储和响应请求</h1>
<p>要给控制面板提供API和访问,我们需要一个新的后端来弥补前面提到的短板,包括下面几个新需求:</p>
<ul>
<li>允许大多数属性的过滤。</li>
<li>允许全文搜索。</li>
<li>支持存储至少30天数据,可以有限度的轮滚。</li>
<li>添加节点即可轻松扩展。</li>
<li>节点失效无影响。</li>
</ul>
<p>而Elasticsearch,是一个可以“准”实时入库、实时请求的搜索引擎。它基于Apache Lucene,由存储索引的节点组成一个分布式高可用的集群。单个节点离线,集群会自动把索引(的分片)均衡到剩余节点上。你可以配置具体每个索引有多少分片,以及这些分片该有多少副本。如果一个主分片离线,就从副本中选一个出来提升为主分片。</p>
<p>Elasticsearch 是面向文档的,某种层度上可以说也是无模式的。这意味着你可以传递任意JSON文档然后就可以索引成字段。对我们的事件来说完全符合要求。</p>
<p>Elasticsearch 同样还有一个非常强大的请求/过滤接口,可以对特定字段搜索,也可以做全文搜索。</p>
<h1 id="elasticsearch-1">事件存入elasticsearch</h1>
<p>有很多工具或者服务可以用来记录事件。我们最终选择了 <a href="http://logstash.net/">Logstash</a>,一个搜集、分析、管理和传输日志的工具。</p>
<p>在内部,通过webhooks推送来的event同时在我们系统的其他部分也有使用,目前我们是用Redis来完成这个功能。Logstash有一个Redis输入插件来从Redis列表里接收日志事件。通过几个小过滤器后,事件通过一个输出插件输出。最常用的输出插件就是 Elasticsearch 插件。</p>
<p>利用 Elasticsearch 丰富的 API 最好的办法就是使用 Kibana,这个工具的口号是“让海量日志有意义”。目前最新的 <a href="http://three.kibana.org/">Kibana 3</a> 是一个纯粹的 JavaScript 客户端版,随后也会成为 Logstash 的默认界面。和之前的版本不同的是,它不在依赖于一个类Logstash模式,而是可以用于任意Elasticsearch索引。</p>
<p><img src="http://9791b61a81187466cf77-03e2fb40b56101ddc8886446c68cb0c1.r77.cf2.rackcdn.com/kibana%20events%202.png" alt="" /></p>
<h1 id="section-2">认证</h1>
<p>到这步,我们已经解决了事件集中的问题,也有了丰富的API来深入解析日志。但是我们不想把所有日志都公开给每个人,所以我们需要一个认证,目前Elasticsearch 和 Kibana 都没提供认证功能,所以寄希望于 Elasticsearch API 是不可能的了。</p>
<p>我们选择了构建双层代理。一层代理用来做认证和流量限速,一层用来转义我们的事件 API 成 Elasticsearch 请求。前面这层代理我们已经以 Apache 2.0 开原协议发布在Github上,叫 <a href="https://github.com/mailgun/vulcan">vulcan</a> 。我们还把我们原来的那套日志 API 也转移到了 Elasticsearch 系统上。</p>
<h1 id="section-3">索引设计</h1>
<p>有很多种方法来确定你如何组织自己的索引,基于文档的数目(每个时间段内),以及查询模式。</p>
<p>Logstash 默认每天创建一个新索引,包括当天收到的全部时间。你可以通过配置修改这个时间,或者采用其他属性来区分索引,比如每个用户一个,或者用事件类型等等。</p>
<p>我们这里每秒有1500个时间,而且我们希望每个账户的轮转时间段都是可配置的。可选项有:</p>
<ul>
<li>一个大索引。</li>
<li>每天一个索引。</li>
<li>每个用户账户一个索引。</li>
</ul>
<p>当然,如果需要的话,这些都可以在未来进一步切分,比如根据事件类型。</p>
<p>管理轮滚的一个办法是在 Elasticsearch 中给每个文档设定 <a href="http://www.elasticsearch.org/guide/reference/mapping/ttl-field/">TTLs</a> 。到了时间 Elasticsearch 就会批量删除过期文档。这种做法使得定制每个账户的轮转时间变得很简单,但是也带来了更多的 IO 操作。</p>
<p>另一个轻量级的办法是直接删除整个索引。这也是 Logstash 默认以天创建索引的原因。过了这天你直接通过 crontab 任务删除索引即可。</p>
<p>不过后面这个办法就没法定制轮转了。我们有很多用户账户,给每个用户每天保持一个索引是不切实际的。当然,给所有用户每天存一个索引又意味着我们要把所有数据都存磁盘上。如果一个账户是保持两天数据的轮转,那么在缓存中的数据就是有限的。在查询多天的垃圾邮件时,处理性能也就受限了。所以,我们需要保留更多的日志以供Kibana访问。</p>
<h1 id="section-4">映射</h1>
<p>为了定义文档(中的字段)如何压缩、索引和存储在索引里,Elasticsearch 有一个叫做 <a href="http://www.elasticsearch.org/guide/reference/mapping/">mapping</a> 的概念。所以为每个字段它都定义了类型,定义了如何分析和标记字段的值以便索引和查询,定义了值是否需要存储,以及其他各种设置。默认的情况,mapping是动态的,也就是说 Elasticsearch 会从它获得的第一个值来尝试猜测字段的类型,然后正式应用这个设置到索引。</p>
<p>如果你的数据来源单一,这样就很好了。但实际可能来源很复杂,或者日志类型根本就不一样,比如我们这,同一个名字的字段的数据类型可能都不一样。 Elasticsearch 会拒绝索引一个类型不匹配的文档,所以我们需要自定义 mapping 。</p>
<p>通过我们的 <a href="http://documentation.mailgun.com/api-events.html">Events API</a> ,我给日志事件的类型定义了一个映射。不是所有的事件都有所有这些字段,不过相同名字的字段肯定是一致的。</p>
<h1 id="section-5">分析器</h1>
<p>默认情况下,字段的 mapping 中就带有 标准分析器。简单的说,就是字符串会被转成小写,然后分割成一个一个单词。然后这些标记化的单词再写入银锁,并指向具体的字段。</p>
<p>有些情况,你可能想要些别的东西来完成不同的效果。比如说账户 ID,电子邮件地址或者网页链接 URL之类的,默认标记器会以斜线分割,而不考虑把整个域名作为一个单独的标记。当你通过 facet 统计域名字段的时候,你得到的会是域名中一段一段标签的细分结果。</p>
<p>要解决这个问题,可以设置索引属性,给对应字段设置成 <code class="highlighter-rouge">not_analyzed</code>。这样在插入索引的时候,这个字段不再经过映射或者标记器。比如对 <code class="highlighter-rouge">domain.name</code> 字段应用这个设置后,每个域名都会完整的作为同一个标签统计 facet 了。</p>
<p>如果你还想在这个字段内通过部分内容查找,你可以使用 <a href="http://www.elasticsearch.org/guide/reference/mapping/multi-field-type/">multi-field type</a>。这个类型可以映射相同的值到不同的核心类型或者属性,然后在不同名称下使用。我们对 IP 地址就使用了这个技术。默认的字段(比如叫<code class="highlighter-rouge">sending-ip</code>)的类型就是 ip,而另一个非默认字段(比如叫 <code class="highlighter-rouge">sending-ip.untouched</code>)则配置成 <code class="highlighter-rouge">not_analyzed</code> 而且类型为字符串。这样,默认字段可以做 IP 地址专有的范围查询,而 <code class="highlighter-rouge">.untouched</code> 字段则可以做 facet 查询。</p>
<p>除此以外,绝大多数字段我们都没用分析器和标记器。不过我们正在考虑未来可以结合上面的多字段类型技巧,应用 <a href="http://www.elasticsearch.org/guide/reference/index-modules/analysis/pattern-capture-tokenfilter/">pattern capture tokenfilter</a> 到某些字段(比如电子邮件地址)上。</p>
<h1 id="section-6">监控</h1>
<p>要知道你的集群怎么样,你就必须要监控它。 Elasticsearch 有非常棒的 API 来获取 <a href="http://www.elasticsearch.org/guide/reference/api/admin-cluster-state/">cluster state</a> 和 <a href="http://www.elasticsearch.org/guide/reference/api/admin-cluster-nodes-stats/">node statistics</a>。我们可以用 <a href="http://graphite.wikidot.com/">Graphite</a> 来存储这些指标并且做出综合表盘,下面就是其中一个面板:</p>
<p><img src="http://9791b61a81187466cf77-03e2fb40b56101ddc8886446c68cb0c1.r77.cf2.rackcdn.com/graphite%20monitoring.png" alt="" /></p>
<p>为了收集这些数据并且传输到 Graphite,我创建了 <a href="http://github.com/mochi/vor">Vör</a>,已经在 <a href="http://www.mochimedia.com/">Mochi Media</a> 下用 MIT/X11 协议开源了。另外一个保证 Redis 列表大小的收集器也在开发中。</p>
<p>除此以外,我们还统计很多东西,比如邮件的收发、点击数,API调用和耗时等等,这些是通过 <a href="https://github.com/etsy/statsd">StatsD</a> 收集的,同样也添加到我们的 Graphite 表盘。</p>
<p><img src="http://9791b61a81187466cf77-03e2fb40b56101ddc8886446c68cb0c1.r77.cf2.rackcdn.com/graphite%20events.png" alt="" /></p>
<p>这绝对是好办法来观察发生了什么。Graphite 有一系列函数可以用来在绘图前作处理,也可以直接返回JSON文档。比如,我们可以很容易的创建一个图片展示 API 请求的数量与服务器负载或者索引速度的联系。</p>
<h1 id="section-7">当前状况</h1>
<p>我们的一些数据:</p>
<ul>
<li>每天大概4kw 到 6kw 个日志事件。</li>
<li>30天轮转一次日志。</li>
<li>30个索引。</li>
<li>每个索引5个分片。</li>
<li>每个分片一个1副本。</li>
<li>每个索引占 2 * 50 到 80 GB空间(因为有副本所以乘2)。</li>
</ul>
<p>为此,我们启动了一共 9 台 Rackspace 云主机,具体配置是这样的:</p>
<ul>
<li>6x 30GB RAM, 8 vCPUs, 1TB disk: Elasticsearch 数据节点。</li>
<li>2x 8GB RAM, 4 vCPUs: Elasticsearch 代理节点, Logstash, Graphite 和 StatsD。</li>
<li>2x 4GB RAM, 2 core: Elasticsearch 代理节点, Vulcan 和 API 服务器</li>
</ul>
<p>大多数主机最终会迁移到专属的平台上,同时保留有扩展新云主机的能力。</p>
<p>Elasticsearch 数据节点都配置了 16GB 内存给 JVM heap。其余都是标准配置。此外还设置了 fieldcache 最大大小为 heap 的 40%,以保证集群不会在 facet 和 sort 内容很多的字段时挂掉。我们同时也增加了一点 <a href="http://www.elasticsearch.org/guide/reference/api/admin-cluster-update-settings/">cluster wide settings</a> 来加速数据恢复和重均衡。另外,相对于我们存储的文档数量来说,<code class="highlighter-rouge">indices.recovery.max_bytes_per_sec</code> 的默认设置实在太低了。</p>
<h1 id="section-8">总结</h1>
<p>我们非常高兴用 Elasticsearch 来保存我们的事件,也得到了试用新 API 和新控制面板中新日志页面的客户们非常积极的反馈。任意字段的可搜索对日志挖掘绝对是一种显著的改善,而 Elasticsearch 正提供了这种高效无痛的改进。当然,Logstash,Elasticsearch 和 Kibana 这整条工具链也非常适合内部应用日志处理。</p>
<p>如果你想了解更多详情或者对我们的 API 有什么疑问,尽管留言。也可以在 Mailgun 博客上<a href="http://blog.mailgun.com/post/new-events-api-detailed-email-tracking-and-search/">阅读更多关于事件 API 的细节</a>。</p>
<p>开心处理日志,开心发送邮件!</p>
用 ElasticSearch 支持 Rexify 网站的搜索功能
2013-09-14T00:00:00+08:00
perl
elasticsearch
rex
http://chenlinux.com/2013/09/14/elasticsearch-for-rexify-website
<p>最近给 Rexify 官网做<a href="http://rex.perl-china.com">中文化</a>工作,除了文字翻译之外,还要负责把服务正常跑起来。网站本身就是一个 <code class="highlighter-rouge">Mojolicious</code> 写的小东西,用 <code class="highlighter-rouge">morbo html/website.pl</code> 命令直接运行就可以监听在 3000 端口,然后通过 nginx 代理发布即可。</p>
<p>不过官网上还有一个高级功能需要另外支持,那就是搜索。</p>
<p>Rexify 官网的搜索功能是通过 ElasticSearch 提供的。这里需要注意一点,官方提供的 <code class="highlighter-rouge">create_index.pl</code> 中,并不是直接把文件内容本身存入 ES 索引的(之前介绍过的 <code class="highlighter-rouge">devopsweekly_index.pl</code> 脚本就是这样做的),而是编码成 Base64 之后再以附件形式存储。</p>
<p>一开始我没注意到这点,结果搜索结果里一直只有标题和链接,没有高亮内容。后来发现是存的 base64 编码后又很疑惑 Rexify 官网是如何把 base64 再解码回来到网页上显示的。幸亏后来想到去 ElasticSearch 官网搜索一下 base64 关键词,然后发现了专门的<a href="https://github.com/elasticsearch/elasticsearch-mapper-attachments">介绍页面</a>。原来是有一个<a href="https://github.com/elasticsearch/elasticsearch-mapper-attachments">插件</a>实现的附件解析,调用了 Apache Tika 库,也就意味着支持 HTML/XML/Office/ODF/PDF/Epub/RTF/TXT/ZIP/MP3/JPG/FLV/Mbox/JAR 等等各种格式的文件。</p>
<p>所以,安装这个插件,然后重建索引,就可以正常提供搜索功能了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>/usr/share/elasticsearch/bin/plugin -install elasticsearch/elasticsearch-mapper-attachments/1.9.0
rexify-website/create_index.pl localhost 9200 html/templates
</code></pre>
</div>
<p>脚本本身超级简单,欢迎大家自行阅读。</p>
Perl 和 Python 的 pack 函数格式字符的区别
2013-09-10T00:00:00+08:00
perl
monitor
moosefs
python
http://chenlinux.com/2013/09/10/different-format-character-of-pack-between-python-and-perl
<p>MooseFS 是运用很广泛的一个分布式文件系统,其自带有一个 python 写的 CGI 页面,可以查看集群状态。不过对于运维来说,这就不太方便纳入 nagios 等其他现有的监控体系中。好在既然它的 CGI 是 python 写的,那么自己照样临摹出一个监控脚本也不是太复杂。</p>
<p>其实整个数据是由 master 的 9421 端口进行 TCP 交互获取的,不过比较麻烦的是并不是普通文本流。CGI 中采用了 pack/unpack 函数来处理 TCP 包。根据数据的前 8 字节确定数据总长度和 MooseFS 的版本,然后依照不同版本的 pack 方式来 unpack 剩余内容。</p>
<p>笔者熟悉 Perl,所以就准备将这个处理流程改用 Perl 完成。结果发现原来 pack/unpack 在 Perl 和 Python 中,写法是不一样的。以 MooseFS 的 info 信息读取代码为例,Python 版如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">s</span> <span class="o">=</span> <span class="n">socket</span><span class="o">.</span><span class="n">socket</span><span class="p">()</span>
<span class="n">s</span><span class="o">.</span><span class="n">connect</span><span class="p">((</span><span class="n">masterhost</span><span class="p">,</span> <span class="n">masterport</span><span class="p">))</span>
<span class="n">mysend</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="n">struct</span><span class="o">.</span><span class="n">pack</span><span class="p">(</span><span class="s">">LL"</span><span class="p">,</span> <span class="mi">510</span><span class="p">,</span> <span class="mi">0</span><span class="p">))</span>
<span class="n">header</span> <span class="o">=</span> <span class="n">myrecv</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="mi">8</span><span class="p">)</span>
<span class="n">cmd</span><span class="p">,</span> <span class="n">length</span> <span class="o">=</span> <span class="n">struct</span><span class="o">.</span><span class="n">unpack</span><span class="p">(</span><span class="s">">LL"</span><span class="p">,</span> <span class="n">header</span><span class="p">)</span>
<span class="k">if</span> <span class="n">cmd</span> <span class="o">==</span> <span class="mi">511</span> <span class="ow">and</span> <span class="n">length</span> <span class="o">==</span> <span class="mi">76</span><span class="p">:</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">myrecv</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="n">length</span><span class="p">)</span>
<span class="n">v1</span><span class="p">,</span> <span class="n">v2</span><span class="p">,</span> <span class="n">v3</span><span class="p">,</span> <span class="n">memusage</span><span class="p">,</span> <span class="n">total</span><span class="p">,</span> <span class="n">avail</span><span class="p">,</span> <span class="n">trspace</span><span class="p">,</span> <span class="n">trfiles</span><span class="p">,</span> <span class="n">respace</span><span class="p">,</span> <span class="n">refiles</span><span class="p">,</span> <span class="n">nodes</span><span class="p">,</span> <span class="n">dirs</span><span class="p">,</span> <span class="n">files</span><span class="p">,</span> <span class="n">chunks</span><span class="p">,</span> <span class="n">allcopies</span><span class="p">,</span> <span class="n">tdcopies</span> <span class="o">=</span> <span class="n">struct</span><span class="o">.</span><span class="n">unpack</span><span class="p">(</span><span class="s">">HBBQQQQLQLLLLLLL"</span><span class="p">,</span> <span class="n">data</span><span class="p">)</span>
<span class="n">ver</span> <span class="o">=</span> <span class="s">'.'</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="nb">str</span><span class="p">(</span><span class="n">v1</span><span class="p">),</span> <span class="nb">str</span><span class="p">(</span><span class="n">v2</span><span class="p">),</span> <span class="nb">str</span><span class="p">(</span><span class="n">v3</span><span class="p">)])</span>
</code></pre>
</div>
<p>而 Perl 版最终写完是这样的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">my</span> <span class="nv">$s</span> <span class="o">=</span> <span class="nn">IO::Socket::</span><span class="nv">INET</span><span class="o">-></span><span class="k">new</span><span class="p">(</span>
<span class="nv">PeerAddr</span> <span class="o">=></span> <span class="nv">$host</span><span class="p">,</span>
<span class="nv">PeerPort</span> <span class="o">=></span> <span class="nv">$port</span><span class="p">,</span>
<span class="nv">Proto</span> <span class="o">=></span> <span class="s">'tcp'</span><span class="p">,</span>
<span class="p">);</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$header</span><span class="p">,</span> <span class="nv">$data</span><span class="p">);</span>
<span class="k">print</span> <span class="nv">$s</span> <span class="nb">pack</span><span class="p">(</span><span class="s">'(LL)>'</span><span class="p">,</span> <span class="mi">510</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
<span class="nb">sysread</span> <span class="nv">$s</span><span class="p">,</span> <span class="nv">$header</span><span class="p">,</span> <span class="mi">8</span><span class="p">;</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$cmd</span><span class="p">,</span> <span class="nv">$length</span><span class="p">)</span> <span class="o">=</span> <span class="nb">unpack</span><span class="p">(</span><span class="s">'(LL)>'</span><span class="p">,</span> <span class="nv">$header</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$cmd</span> <span class="o">==</span> <span class="mi">511</span> <span class="ow">and</span> <span class="nv">$length</span> <span class="o">==</span> <span class="mi">76</span> <span class="p">)</span> <span class="p">{</span>
<span class="nb">sysread</span> <span class="nv">$s</span><span class="p">,</span> <span class="nv">$data</span><span class="p">,</span> <span class="nv">$length</span><span class="p">;</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$v1</span><span class="p">,</span> <span class="nv">$v2</span><span class="p">,</span> <span class="nv">$v3</span><span class="p">,</span> <span class="nv">$memusage</span><span class="p">,</span> <span class="nv">$total</span><span class="p">,</span> <span class="nv">$avail</span><span class="p">,</span> <span class="nv">$trspace</span><span class="p">,</span> <span class="nv">$trfiles</span><span class="p">,</span> <span class="nv">$respace</span><span class="p">,</span> <span class="nv">$refiles</span><span class="p">,</span> <span class="nv">$nodes</span><span class="p">,</span> <span class="nv">$dirs</span><span class="p">,</span> <span class="nv">$files</span><span class="p">,</span> <span class="nv">$chunks</span><span class="p">,</span> <span class="nv">$allcopies</span><span class="p">,</span> <span class="nv">$tdcopies</span><span class="p">)</span> <span class="o">=</span> <span class="nb">unpack</span><span class="p">(</span><span class="s">'(SCCQQQQLQLLLLLLL)>'</span><span class="p">,</span> <span class="nv">$data</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$ver</span> <span class="o">=</span> <span class="s">"$v1.$v2.$v3"</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<p>不同处主要有两点:</p>
<ol>
<li>关于 <code class="highlighter-rouge">big-endian</code> 定义的 <code class="highlighter-rouge">></code> 符号位置不同,Python 里写在起首一次性全部生效;Perl 里需要每个格式符单独定义,或者采用括号合起来总定义;</li>
<li>Python 里的 <code class="highlighter-rouge">H</code> 格式符表示 <code class="highlighter-rouge">unsigned short</code>,在 Perl 里应该是 <code class="highlighter-rouge">S</code>;Python 里的 <code class="highlighter-rouge">B</code> 格式符表示 <code class="highlighter-rouge">unsigned char</code>,在 Perl 里应该是 <code class="highlighter-rouge">C</code>。</li>
</ol>
<p>翻看了一下,在 PHP 和 Ruby 中,格式符定义和 Perl 是一样的,不清楚为什么 Python 这么特殊==!</p>
<p>各语言关于 <code class="highlighter-rouge">pack</code> 格式符的文档链接如下:</p>
<ol>
<li><a href="http://www.w3school.com.cn/php/func_misc_pack.asp">http://www.w3school.com.cn/php/func_misc_pack.asp</a></li>
<li><a href="http://www.kuqin.com/rubycndocument/man/pack_template_string.html">http://www.kuqin.com/rubycndocument/man/pack_template_string.html</a></li>
<li><a href="http://docs.python.org/2/library/struct.html">http://docs.python.org/2/library/struct.html</a></li>
<li><a href="http://perldoc.perl.org/functions/pack.html">http://perldoc.perl.org/functions/pack.html</a></li>
</ol>
编译最新 3.10 内核在 RHEL6 上支持 Docker
2013-08-27T00:00:00+08:00
cloud
linux
http://chenlinux.com/2013/08/27/using-docker-on-rhel6-the-hard-way
<p>之前在 Fedora19 上试图自己通过编译 3.10 内核的方式来完成 aufs 的支持,但是一直有问题,哪怕同样的步骤,github 上其他人都可以,只能怀疑是我个人电脑问题了。不过后来通过 SPEC 方式完成了最终测试,感谢 sciurus 童鞋的<a href="https://github.com/sciurus/docker-rhel-rpm">项目</a>。</p>
<p>部署过程如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># 安装这个包以便使用 mock 命令在 chroot 环境下打包</span>
yum install -y fedora-packager
<span class="c"># 下载我的而不是原作者的,因为里面 aufs 和 lxc 的下载链接都已经更新了,原来的404了</span>
git clone https://github.com/chenryn/docker-rhel-rpm.git
spectool -g -C docker docker/docker.spec
mock -r epel-6-x86_64 --buildsrpm --spec docker/docker.spec --sources docker --resultdir output
mock -r epel-6-x86_64 --rebuild --resultdir output output/docker-0.6.0-1.el6.src.rpm
spectool -g -C lxc lxc/lxc.spec
mock -r epel-6-x86_64 --buildsrpm --spec lxc/lxc.spec --sources lxc --resultdir output
mock -r epel-6-x86_64 --rebuild --resultdir output output/lxc-0.8.0-3.el6.src.rpm
spectool -g -C kernel-ml-aufs kernel-ml-aufs/kernel-ml-aufs-3.10.spec
mock -r epel-6-x86_64 --buildsrpm --spec kernel-ml-aufs/kernel-ml-aufs-3.10.spec --sources kernel-ml-aufs --resultdir output
mock -r epel-6-x86_64 --rebuild --resultdir output output/kernel-ml-aufs-3.10.5-1.el6.src.rpm
<span class="nb">cd </span>output
yum localinstall --nogpgcheck kernel-ml-aufs-3.10.5-1.el6.x86_64.rpm lxc-0.8.0-3.el6.x86_64.rpm lxc-libs-0.8.0-3.el6.x86_64.rpm docker-0.6.0-1.el6.x86_64.rpm
<span class="nb">echo</span> <span class="s1">'none /sys/fs/cgroup cgroup defaults 0 0'</span> > /etc/fstab
reboot
</code></pre>
</div>
<p>kernel 文件来自 RHEL,不过我试了下,在我的Fedora19上也正常可用。3.10.5 和 3.2 相比,第一 3.10 将会是未来一段时间内 kernel 的主线支持;第二 docker 官方说在 3.8 之前有点小 bug 可能会被触发。</p>
在 Docker 上运行 PerlDancer 示例
2013-08-26T00:00:00+08:00
cloud
perl
http://chenlinux.com/2013/08/26/running-perldancer-on-docker
<p>搭建好了 docker 环境后,就可以来试试用 docker 跑一个应用实例来看看了。和 Vagrant 比较类似,docker 也是用一个配置文件来规划其基础镜像内的部署,不过值得注意的是,在 <code class="highlighter-rouge">Dockerfile</code> 里的每一个指令成功执行后,docker 默认都会 commit 一次,这样就节省了一些空间和时间。</p>
<p>构建失败的镜像,在 <code class="highlighter-rouge">docker images</code> 命令输出中显示为 <code class="highlighter-rouge"><none></code> 可以根据具体的 commit id,调用 <code class="highlighter-rouge">docker rmi <id></code> 命令清除。</p>
<p>一个比较简单的 <code class="highlighter-rouge">Dockerfile</code> 示例是这样的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="no">FROM</span> <span class="n">centos</span><span class="p">:</span><span class="mi">6</span><span class="o">.</span><span class="mi">4</span>
<span class="no">RUN</span> <span class="n">yum</span> <span class="n">install</span> <span class="n">make</span> <span class="n">gcc</span> <span class="n">wget</span> <span class="n">perl</span> <span class="n">perl</span><span class="o">-</span><span class="n">devel</span> <span class="n">perl</span><span class="o">-</span><span class="no">Time</span><span class="o">-</span><span class="no">HiRes</span> <span class="n">perl</span><span class="o">-</span><span class="no">CGI</span> <span class="n">perl</span><span class="o">-</span><span class="n">libwww</span><span class="o">-</span><span class="n">perl</span> <span class="n">perl</span><span class="o">-</span><span class="no">Module</span><span class="o">-</span><span class="no">Build</span> <span class="n">perl</span><span class="o">-</span><span class="no">Test</span><span class="o">-</span><span class="no">Simple</span> <span class="n">perl</span><span class="o">-</span><span class="no">Test</span><span class="o">-</span><span class="no">Deep</span> <span class="n">perl</span><span class="o">-</span><span class="no">YAML</span>
<span class="no">RUN</span> <span class="n">wget</span> <span class="n">http</span><span class="ss">:/</span><span class="o">/</span><span class="n">cpanmin</span><span class="p">.</span><span class="nf">us</span>
<span class="no">RUN</span> <span class="n">perl</span> <span class="n">cpanm</span> <span class="no">Dancer</span>
<span class="no">ADD</span> <span class="sr">/var/</span><span class="n">www</span><span class="o">/</span><span class="n">dancerapp</span> <span class="n">app</span>
<span class="no">EXPOSE</span> <span class="mi">3000</span>
<span class="no">CMD</span> <span class="n">perl</span> <span class="n">app</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">app</span><span class="p">.</span><span class="nf">pl</span>
</code></pre>
</div>
<p>然后运行如下命令构建镜像:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>docker build -t chenryn/perldancer
</code></pre>
</div>
<p>如果构建都成功的话,那就是正式运行了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>docker run -p 8080:3000 -d chenryn/perldancer
</code></pre>
</div>
<p>运行起来以后,可以通过 <code class="highlighter-rouge">docker ps</code> 命令看到本机上运行着的容器状态信息。同样,也可以通过映射的 8080 端口访问到页面了。</p>
<p>正在测试通过 <code class="highlighter-rouge">plenv</code> 来使用高版本的 perl,目前比较郁闷的是因为 <code class="highlighter-rouge">plenv</code> 是通过 <code class="highlighter-rouge">~/.profile</code> 来在每次登陆的时候自动切换到指定版本的,而 <code class="highlighter-rouge">docker</code> 里的 <code class="highlighter-rouge">RUN</code> 调用 <code class="highlighter-rouge">/bin/sh -c</code> 不会调用到这些文件,所以一直还是使用系统自带版本。而在 <code class="highlighter-rouge">RUN</code> 指令里每行都写一个 <code class="highlighter-rouge">source $HOME/.profile</code> 也很难看的。</p>
快速在 CentOS6 上运行 docker
2013-08-24T00:00:00+08:00
cloud
http://chenlinux.com/2013/08/24/using-docker-on-centos6
<p>docker 是由著名 PAAS 公司 dotcloud 开源的 linux 容器项目,在此之前,只有 cloudfoundry 下属的 warden 半死不活的慢慢前进着。</p>
<p>不管是 docker 还是 warden,其原理大多是通过 LXC( 即 CGroup 和 namespace 的结合)以及 AUFS 的结合,完成比较彻底的容器虚拟化。这里有个问题:AUFS 不是 linux 官版内核支持的文件系统。所以到现在,各种 PAAS 都是运行在 Ubuntu 系统上,因为只有这个系列的发行版默认打了 AUFS 的补丁。这也严重影响了 PAAS 开源社区的扩容:</p>
<ol>
<li>RedHat 发行版系列才是企业用户最多的 linux 发行版;</li>
<li>Debian 社区已经宣布在未来会放弃默认打 AUFS 补丁的做法。</li>
</ol>
<p>docker 目前已经在积极准备将代码 port 到 BtrFS 上以备未来,不过在此之前,我们还是可以通过自己打补丁的方式,在 RedHat 系列上尝试 docker 的。目前社区已经有很多尝试:</p>
<ol>
<li><a href="http://neh.al/?p=1">Installing Docker on Fedora 18</a></li>
<li><a href="http://blog.rage.net/2013/08/04/installing-docker-on-centos-6/">Installing Docker on Centos 6</a></li>
<li><a href="https://github.com/sciurus/docker-rhel-rpm">files needed to build RPMs for the dependencies of docker</a></li>
<li><a href="https://github.com/failshell/chef-docker">chef-docker</a></li>
<li><a href="http://nareshv.blogspot.in/2013/08/installing-dockerio-on-centos-64-64-bit.html">Installing Dockerio on Centos6.4-64bit</a></li>
</ol>
<p>其中,包括有三种内核,源代码编译支持3.8的,spec编译支持3.10的,以及已经打包完成的3.2的。</p>
<p>我已经尝试过在 Fedora19 上通过源代码编译,似乎内核从3.8到3.10有些变化,编译失败了。(但是尝试过编译3.8的确实没问题)</p>
<p>下面通过最简单的已经打包完成的3.2内核来快速部署 docker 到 CentOS6 上,以便尝鲜:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>rpm -e kernel-firmware
rpm -i http://get.docker.io/kernels/kernel-3.2.40_grsec_dotcloud-4.x86_64.rpm
/sbin/dracut --add-drivers dm-mod --add-drivers linear <span class="s2">""</span> 3.2.40-grsec-dotcloud
grub-install /dev/sda1
<span class="nb">echo</span> <span class="s2">"blacklist evbug"</span> >> /etc/modprobe.d/blacklist.conf
<span class="nb">echo</span> <span class="s2">"kernel.grsecurity.chroot_caps = 0"</span> >> /etc/sysctl.conf
<span class="nb">echo</span> <span class="s2">"sysctl kernel.grsecurity.chroot_caps=1"</span> >> /etc/rc.local
<span class="nb">echo</span> <span class="s2">"net.ipv4.ip_forward = 1"</span> >> /etc/sysctl.conf
mkdir /cgroup
<span class="nb">echo</span> <span class="s2">"none /cgroup cgroup defaults 0 0"</span> >> /etc/fstab
cat >> /boot/grub/grub.conf<span class="sh"><<EOF
title CentOS (3.2.40_grsec_dotcloud-4.x86_64)
root (hd0,0)
kernel /boot/vmlinuz-3.2.40-grsec-dotcloud ro root=LABEL=/ rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM selinux=0
initrd /boot/initramfs-3.2.40-grsec-dotcloud.img
EOF
</span>reboot
</code></pre>
</div>
<p>内核的更新就是这些,记住这个包不支持 selinux,所以启动项里要加上 <code class="highlighter-rouge">selinux=0</code>。</p>
<p>然后重启登录重启并选择了新内核的主机,继续安装一些依赖工具:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget <span class="s2">"ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home%3A/awk2007%3A/fixes/Fedora_17/src/aufs-util-9999-14.1.src.rpm"</span>
sudo yum install glibc-static
rpmbuild --rebuild aufs-util-9999-14.1.src.rpm
rpm -U /root/rpmbuild/RPMS/x86_64/aufs-util-9999-14.1.x86_64.rpm
wget ftp://ftp.univie.ac.at/systems/linux/dag/redhat/el6/en/x86_64/dag/RPMS/lxc-0.8.0-1.el6.rf.x86_64.rpm
wget http://apt.sw.be/redhat/el6/en/x86_64/dag/RPMS/lxc-libs-0.8.0-1.el6.rf.x86_64.rpm
rpm -U lxc-0.8.0-1.el6.rf.x86_64.rpm lxc-libs-0.8.0-1.el6.rf.x86_64.rpm
</code></pre>
</div>
<p>然后下载 docker 的二进制文件运行,用源代码的话比较麻烦,docker 是用 golang 写的……</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget http://get.docker.io/builds/Linux/x86_64/docker-latest.tgz
tar xzf docker-latest.tgz
<span class="nb">cd </span>docker-latest
</code></pre>
</div>
<p>启动 docker 进程,输出如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[root@localhost docker-latest]# ./docker -d &
2013/08/24 18:24:18 WARNING: You are running linux kernel version 3.2.40-grsec-dotcloud, which might be unstable running docker. Please upgrade your kernel to 3.8.0.
2013/08/24 18:24:18 Listening for HTTP on /var/run/docker.sock (unix)
</code></pre>
</div>
<p>然后就可以通过 docker 命令运行了,示例及输出如下所示:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[root@localhost docker-latest]# ./docker run -i -t busybox /bin/sh
2013/08/24 18:24:30 POST /v1.4/containers/create
2013/08/24 18:24:30 POST /v1.4/images/create?fromImage=busybox&tag=
Pulling repository busybox
Pulling image e9aa60c60128cad1 (latest) from busybox
Pulling e9aa60c60128cad1 metadata
Pulling e9aa60c60128cad1 fs layer
Downloading 2.284 MB/2.284 MB (100%)
2013/08/24 18:28:37 POST /v1.4/containers/create
2013/08/24 18:28:37 POST /v1.4/containers/cdf0feaf24a9/start
2013/08/24 18:28:37 POST /v1.4/containers/cdf0feaf24a9/resize?h=27&w=121
2013/08/24 18:28:37 POST /v1.4/containers/cdf0feaf24a9/attach?logs=1&stderr=1&stdin=1&stdout=1&stream=1
BusyBox v1.19.3 (Ubuntu 1:1.19.3-7ubuntu1.1) built-in shell (ash)
Enter 'help' for a list of built-in commands.
/ #
/ # ls
bin dev etc lib lib64 proc sbin sys tmp usr
/ # cd /root
/bin/sh: cd: can't cd to /root
</code></pre>
</div>
<p>可以看到,现在登录进来是不能切换目录到 root 家目录的。</p>
<p>docker 已经运行起来了,更多实例,就可以看着 docker.io 上的<a href="http://docs.docker.io/en/latest/examples/">文档</a>慢慢进行了</p>
BeiJing Perl Workshop 2013 参会总结
2013-08-14T00:00:00+08:00
perl
http://chenlinux.com/2013/08/14/my-perl-workshop-experience
<p>上周六在万通会议中心参加了 BeiJing Perl Workshop 2013 ,并做了 40 分钟长的关于 ElasticSearch 的演讲。上届 2011 作为一个看客,两年后作为一个积极参与和演讲者,真的有必要记录一下。</p>
<p>一天中最让大家惊奇和感兴趣的无疑是胡松涛带来的 3D 打印机以及每人都有的 Perl 小挂件 —— 没错,就是用 3D 打印出来的小东西。</p>
<p><img src="/images/uploads/3d_perlchina.jpg" alt="large" /></p>
<p><img src="/images/uploads/3d_perl.jpg" alt="small" /></p>
<p>最遗憾的是来自alibaba的两位演讲者因为公司问题临时退出。</p>
<p>DeNA 的演讲者是一位由程序员转职的产品经理,明显演讲的技巧水平是要高过我们这些纯码农的,节奏控制完全值得学习。<br />
不知道对于其他 perler 来说,测试是否会主动去做,但是我是蛮习惯使用 <code class="highlighter-rouge">Dancer::Test</code> 的,包括其他项目用 <code class="highlighter-rouge">Test::More</code>,我觉得都是蛮好的习惯。所以对董余康在演讲中比较重点的提到测试的便捷我是比较有爱的,但是从 QQ 群上的反应来看,大家更期待的功能和开发便捷上的介绍?<br />
总的来说,演讲题目偏于 PerlDancer 的 hello world,我个人本身对 Dancer 有一定了解,所以内容上没太注意。如果以后发现大家对 <code class="highlighter-rouge">Dancer</code> 有进一步的兴趣,我考虑可以在 YY 频道上介绍一下用 <code class="highlighter-rouge">Dancer::Plugin</code> 实现一个自己喜欢的keyword?</p>
<p>扶凯的 <code class="highlighter-rouge">Mojolicious</code> 演讲应该就比较符合大众的期待。我是不太喜欢单独定义 route 的方式,不管是 RoR 还是 mojo,除此以外,mojo 另一个不错的就是 <code class="highlighter-rouge">TagHelper</code> 了。我觉得将 <code class="highlighter-rouge">Dancer</code> 和 <code class="highlighter-rouge">Mojolicious::Lite</code> 相比是不厚道的,<code class="highlighter-rouge">Dancer</code> 也可以多文件使用,<code class="highlighter-rouge">dancer -a web_app</code> 命令就生成了完整的项目层次。<br />
关于 mojo 的演讲,牛氓同学说没有关于 <code class="highlighter-rouge">Mojo</code> 的 OO 实现的内容,我也觉得比较遗憾,不过这个内容是否适合在面对上百听众的时候分享,也是一个问题?</p>
<p>李瑞彬演讲中提到的 <code class="highlighter-rouge">cpanspec</code> 和 <code class="highlighter-rouge">yum search perl(LWP::Simple)</code> 两个小技巧很不错~</p>
<p>刘刊和大家不同,采用了直接 code 解读和 shell 演示的办法介绍了他的 <code class="highlighter-rouge">pantheon</code> 里面的一些思想和简单用法。其实我去年就见过这个的使用,不过似乎到现在依然没有完整的文档?作为一个在雅虎超大环境下经过实践的自动化运维平台项目,如果配上完善的文档,应该可以成为一个可以对 Perl 普及有很不错推动力的好项目。<strong>真心希望刘刊和 <code class="highlighter-rouge">pantheon</code> 的其他使用者可以花点时间,整理一份快速入手的 cookbook,迁移代码到github,独立域名发布,完成从内部项目到社区项目的转身~</strong><br />
目前只能通过 CPAN 安装,安装很方便,但是没人演示教导,真的不知道怎么用……</p>
<p>也来说说我自己的演讲。这个话题其实比较尴尬,前三分之一介绍 <code class="highlighter-rouge">ElasticSearch</code>,中间还有一部分是 <code class="highlighter-rouge">logstash</code>,都是 Perl 无关的内容。演讲完后其实发现很多人依然不清楚到底是可以用来干吗……或许我直接只讲 <code class="highlighter-rouge">Message::Passing</code> ,每个插件如何用,效果应该更好一些吧?唯一高兴的,是演讲刚好控制到了40分钟内讲完。</p>
<p>许大师提到一个大计划,要把 <code class="highlighter-rouge">AnyEvent</code> 和 <code class="highlighter-rouge">nginx</code> 无缝结合。这个好是好,能不能出来是另一回事……</p>
<p>闪电演讲依然火爆,不过我有事情先走了,不知道后来还有几位超时被敲锣的~~<br />
对了,第一个闪电提到的 <code class="highlighter-rouge">$obj->can($func_name)->()</code> 这个用法很帅,记录一下。</p>
<p>原本在 CU 上答应网友在闪电上分享一下 <code class="highlighter-rouge">autobox</code> 的用法,也没法做了,有点违约的小羞愧。以后也争取在 YY 频道上说说。</p>
<p>作为演讲者,还另外免费领了一本《HBase管理指南》,实在其他几本 Perl 的都已经有了——而且大多数演讲者都是~~</p>
<p>BJPW 之后第二天就开始 YAPC::EU,国外大神基本都在欧洲讨论 Future Perl,北京的朋友们自娱自乐也还是比较成功了~</p>
<p>最后吐槽一个,全天没有 Perl6 的演讲,结果文化衫上是 Perl6 的蝴蝶。。。其实如果能让广州那个 <code class="highlighter-rouge">MoarVM</code> 的开发者叫来讲讲也挺好的。</p>
<p><img src="/images/uploads/T-shirt.jpg" alt="大会T恤衫" /></p>
<p>附另外两位同仁的大会感受博文:</p>
<ol>
<li><a href="http://www.weibo.com/alickzhao">@赵涛Alick</a> 的 《<a href="http://wp-awesome.rhcloud.com/2013/08/18/perl-china-2013-notes/">Perl China 2013 活动后记</a>》</li>
<li>Aka.Why 的 《<a href="http://blog.aka-cool.net/blog/2013/08/12/learn-from-beijing-perl-workshop-2013/">参加Beijing Perl Workshop 2013后感</a>》</li>
</ol>
Selenium 测试框架介绍
2013-07-22T00:00:00+08:00
javascript
firefox
perl
automation
http://chenlinux.com/2013/07/22/intro-selenium-remote-driver
<p>Selenium 是一个自动化网站测试框架,包括 IDE、WebDriver 和 Grid 三个套件。其官网地址见:<a href="http://docs.seleniumhq.org/projects/">http://docs.seleniumhq.org/projects/</a>。其中 Grid 用以跨主机的集群测试,今天就不讲了。而 WebDriver 则是用以控制 Selenium Server(Server 上可以接受并启动的浏览器包括Firefox、IE、Chrome、Safari、Android、IPhone、PhantomJS 等等)进行具体测试动作的客户端,其早期版本叫做 Remote Control。</p>
<p>最有特色和帮助的,是 IDE 部分,这是一个 Firefox 的 xpi 插件。通过下载安装,就可以启用,然后就是最简单不过的浏览器操作录制,结束动作后就可以自动导出各种支持的语言版本的 WebDriver 程序。</p>
<p>注意在安装好 xpi 后,在 IDE 上并不能同步看到生成的程序内容,并不是说没有录制,而是默认不显示 <code class="highlighter-rouge">options/format</code> 的内容。在 <code class="highlighter-rouge">options/options</code> 里把 <code class="highlighter-rouge">active developer tools</code> 选项激活就可以了。</p>
<p>Selenium 是一个 java 项目,官方支持的客户端程序包括 Java、C#、Ruby 和 Python2。社区支持的包括 Perl、PHP 和 Haskell 等等。</p>
<p>注意 Selenium 的 WebDriver 和 Remote Control 两个版本之间 API 已经完全不一样,所以在 IDE 录制的时候,format 已经要选 WebDriver API 的才能用——除非你还找得到老版本的 Selenium Server,反正我是没找到。</p>
<p>不巧的是目前官网上的插件列表中,只有官方支持的四个更新了 WebDriver 的 IDE 支持。所以直接从官网上安装的 Perl plugin 其实是没用的。不过不要紧,我很容易就找到了支持 WebDriver 的 Perl 模块,并且还使用 Perl 模块完成了对 Selenium Server 的管理。</p>
<p>这里要用到两个 CPAN 模块:<a href="https://metacpan.org/module/Selenium::Server">Selenium::Server</a> 和 <a href="https://metacpan.org/module/Selenium::Remote::Driver">Selenium::Remote::Driver</a>。</p>
<p>由于 Firefox addons 网站上的 <a href="https://addons.mozilla.org/zh-CN/firefox/addon/selenium-ide-perl-formatter/?src=search">Selenium IDE: Perl Formatter</a> 还是老版本的,即 <a href="https://metacpan.org/module/Test::WWW::Selenium">Test::WWW::Selenium</a> 配套的,所以我们需要自行安装新版本插件。</p>
<p>新版本插件也就是一段 javascript 代码,在 <a href="https://metacpan.org/module/Selenium::Remote::Driver">Selenium::Remote::Driver</a> 代码库目录中已经存在,即 <a href="https://github.com/aivaturi/Selenium-Remote-Driver/blob/master/ide-plugin.js">https://github.com/aivaturi/Selenium-Remote-Driver/blob/master/ide-plugin.js</a>。</p>
<p>按照 js 文件开头注释中的介绍,在 Selenium IDE 的 <code class="highlighter-rouge">options/options</code> 菜单的 <code class="highlighter-rouge">Formats</code> 选项卡上点击 <code class="highlighter-rouge">Add</code> 按钮,给新的 format 取名为 <code class="highlighter-rouge">Perl-WebDriver</code>,然后把整个 js 文件内容贴进文本框内保存即可。</p>
<p>现在,录制操作只需选择使用 <code class="highlighter-rouge">Perl-WebDriver</code> 格式,就可以生成 Perl 测试脚本使用了。</p>
<p>下一个问题,就是 Selenium Server 的运行。IDE 生成的脚本只负责连接 server 并发送命令。server 的 状况在 IDE 中是在 <code class="highlighter-rouge">options/formats</code> 中定义的变量,即 Selenium RC host 、Selenium RC port 和 environment。默认是 <code class="highlighter-rouge">localhost</code>、<code class="highlighter-rouge">4444</code> 和 <code class="highlighter-rouge">firefox</code>。在生成脚本的时候会自动替换。</p>
<p>也就是说,我们需要自己部署程序,再运行一个脚本,启用 java 程序,来运行 Selenium Server。</p>
<p>这里就可以用上 <a href="https://metacpan.org/module/Selenium::Server">Selenium::Server</a> 了。程序的下载、启用、参数配置和停止,都有该模块完成。</p>
<p>最后一步,我们可以把 <a href="https://metacpan.org/module/Selenium::Server">Selenium::Server</a> 的相关代码,也贴进 IDE 的 <code class="highlighter-rouge">options/formats</code> 的 <code class="highlighter-rouge">Header</code> 和 <code class="highlighter-rouge">Footer</code> 模板里。这样不用每次自己粘贴了——自己粘贴代码还不如直接自己启用一个固定监听 4444 端口的 java 程序得了。</p>
<p>IDE 截图如下:</p>
<p><img src="/images/uploads/selenium-ide.png" alt="selenium-ide" /></p>
<p>生成脚本如下所示:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Selenium::</span><span class="nv">Server</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Selenium::Remote::</span><span class="nv">Driver</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Test::</span><span class="nv">More</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$server</span> <span class="o">=</span> <span class="nn">Selenium::</span><span class="nv">Server</span><span class="o">-></span><span class="k">new</span><span class="p">;</span>
<span class="nv">$server</span><span class="o">-></span><span class="nv">start</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$driver</span> <span class="o">=</span> <span class="nn">Selenium::Remote::</span><span class="nv">Driver</span><span class="o">-></span><span class="k">new</span><span class="p">(</span>
<span class="nv">remote_server_addr</span> <span class="o">=></span> <span class="nv">$server</span><span class="o">-></span><span class="nv">host</span><span class="p">,</span>
<span class="nv">port</span> <span class="o">=></span> <span class="nv">$server</span><span class="o">-></span><span class="nv">port</span>
<span class="p">);</span>
<span class="nv">$driver</span><span class="o">-></span><span class="nv">get</span><span class="p">(</span><span class="s">"http://10.2.21.100:8081/?results=88ceefac3c0c588d14f579d0c47f74fc"</span><span class="p">);</span>
<span class="nv">$driver</span><span class="o">-></span><span class="nv">find_element</span><span class="p">(</span><span class="s">"DNS可用性测试"</span><span class="p">,</span> <span class="s">"link"</span><span class="p">)</span><span class="o">-></span><span class="nv">click</span><span class="p">;</span>
<span class="nv">like</span><span class="p">(</span><span class="sx">qr/^[\s\S]*各地测试可用性[\s\S]*$/</span><span class="p">,</span><span class="nv">$driver</span><span class="o">-></span><span class="nv">find_element</span><span class="p">(</span><span class="s">"BODY"</span><span class="p">,</span> <span class="s">"css"</span><span class="p">)</span><span class="o">-></span><span class="nv">get_text</span><span class="p">);</span>
<span class="nv">$driver</span><span class="o">-></span><span class="nv">quit</span><span class="p">();</span>
<span class="nv">done_testing</span><span class="p">();</span>
<span class="nv">$server</span><span class="o">-></span><span class="nv">stop</span><span class="p">;</span>
</code></pre>
</div>
<p>脚本中这个 <code class="highlighter-rouge">click</code> 操作显然是直接根据动作录制的,那么 <code class="highlighter-rouge">find_element()->get_text</code> 是怎么来的呢?其实 Selenium IDE 已经修改了浏览器内鼠标右键菜单的选项。在选中的任意网页元素上单击鼠标右键,菜单中就有 <code class="highlighter-rouge">Show All Available Commands</code> 子菜单,只需要选择就可以了。方便吧!</p>
<p>生成的脚本直接运行,就可以完成测试了。</p>
<p>和 <code class="highlighter-rouge">Selenium</code> 类似的,还有 <a href="https://metacpan.org/module/WWW::WebKit">WWW::WebKit</a> 模块,它是调用 <a href="https://metacpan.org/module/Gtk3::WebKit">Gtk3::WebKit</a> 作为后端浏览器支持,不过经过我个人电脑测试,要安装好 <a href="https://metacpan.org/module/Gtk3::WebKit">Gtk3::WebKit</a> 本身就是一件很复杂的事情。加上有时候我们也需要比较不同浏览器的效果是不是有所不同。所以,还是用 Selenium 吧。</p>
<p>注:在最近一期 PerlWeekly 对 Perl 社区创业公司 Lokku/Nestoria 的<a href="http://blogs.perl.org/user/ovid/2013/07/perl-startups-lokkunestoria.html">访谈</a>中,Lokku 公司 CTO,Alex Balhatchet 也提到准备使用 Selenium 改造公司的自动化测试。</p>
<p>补:刚发现 Selenium 的 PHP 客户端,是 Facebook 写的。</p>
<p><strong>2013 年 07 月 25 日补</strong></p>
<p><code class="highlighter-rouge">Selenium</code> 的另一个功能是自己插入 <code class="highlighter-rouge">javascript</code> 到页面里执行。比如我们可以利用 HTML5 的 <code class="highlighter-rouge">WebTiming</code> 特性测试页面的下载时间:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">my</span> <span class="nv">$webtiming</span> <span class="o">=</span> <span class="sx">q{
var performance = window.performance
|| window.webkitPerformance
|| window.mozPerformance
|| window.msPerformance
|| {};
var timings = performance.timing || {};
return timings;
}</span><span class="p">;</span>
<span class="nv">$driver</span><span class="o">-></span><span class="nv">get</span><span class="p">(</span><span class="s">"http://stackoverflow.com/"</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$res</span> <span class="o">=</span> <span class="nv">$driver</span><span class="o">-></span><span class="nv">execute_script</span><span class="p">(</span><span class="nv">$webtiming</span><span class="p">);</span>
<span class="k">for</span> <span class="p">(</span> <span class="nb">sort</span> <span class="nb">keys</span> <span class="nv">%$res</span> <span class="p">)</span> <span class="p">{</span>
<span class="nb">printf</span> <span class="s">"%s %s\n"</span><span class="p">,</span> <span class="nv">$_</span><span class="p">,</span> <span class="nv">$res</span><span class="o">-></span><span class="p">{</span><span class="nv">$_</span><span class="p">}</span><span class="o">/</span><span class="mi">1000</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<p><code class="highlighter-rouge">WebTiming</code> 详细列出了每个阶段的时间。如果 js 写的好,可以写具体某个点触发,就更好了。</p>
<p><strong>2013 年 07 月 26 日补</strong></p>
<p><code class="highlighter-rouge">Selenium::Remote::Driver</code> 只发送操作命令到远端服务器,不具有操作本地浏览器功能。所以无法像 Ruby 的 <code class="highlighter-rouge">Selenium::WebDriver</code> 那样控制本地浏览器,甚至包括插入 <code class="highlighter-rouge">.xpi</code> 插件到自定义的 profile 里完成更复杂的功能:比如用 <code class="highlighter-rouge">Firebug</code>。有一个 Ruby 模块叫 <a href="https://github.com/jfirebaugh/capybara-firebug">capybara-firebug</a>,就是利用这个办法扩展了 <code class="highlighter-rouge">capybara</code> 测试框架。</p>
【Logstash 系列】根据事件统计值报警
2013-07-11T00:00:00+08:00
logstash
ruby
http://chenlinux.com/2013/07/11/howto-filter-count-in-logstash
<p>之前已经用很多博文说过了 logstash 如何配合 elasticsearch 以及 kibana 来做日子分析和实时搜索。其实 logstash 上百个插件还有很多其他的玩法,绝不是局限在日志搜索统计方面的。今天就展示另一个做法。根据日志中的异常值出现频率报警。</p>
<p>在 logstash 的官网上,针对这个问题采用的办法是讲异常值计数 output 到 <code class="highlighter-rouge">statsd</code> 中,然后可以用通过观测 <code class="highlighter-rouge">graphite</code> 图形变化来判断异常。(或者配合 nagios 的 <code class="highlighter-rouge">check_graphite</code> 插件?) 官网说明见:<a href="http://logstash.net/docs/1.1.13/tutorials/metrics-from-logs">http://logstash.net/docs/1.1.13/tutorials/metrics-from-logs</a></p>
<p>如果不想一直盯着页面看的话,可以利用另外几个插件来实现类似的做法,比如我要监控访问日志,如果其中 504 状态码<del>每分钟</del>超过 100 次,就报警出来。logstash 配置如下:</p>
<p><strong>2014 年 08 月 20 日注:上面说法有误,<code class="highlighter-rouge">rate_1m</code> 的含义是:最近 1 分钟内的每秒速率!</strong></p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">input</span> <span class="p">{</span>
<span class="n">stdin</span> <span class="p">{</span>
<span class="n">type</span> <span class="o">=></span> <span class="s2">"apache"</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">filter</span> <span class="p">{</span>
<span class="n">grok</span> <span class="p">{</span>
<span class="n">pattern</span> <span class="o">=></span> <span class="s2">"</span><span class="se">\[</span><span class="s2">%{HTTPDATE:ts}</span><span class="se">\]</span><span class="s2"> %{NUMBER:status} %{IPORHOST:remotehost} %{URIHOST} %{WORD} %{URIPATHPARAM:url} HTTP/%{NUMBER} %{URIHOST:oh} %{NUMBER:responsetime:float} %{NUMBER:upstreamtime:float} (?:%{NUMBER:bytes:float}|-)"</span>
<span class="n">type</span> <span class="o">=></span> <span class="s2">"apache"</span>
<span class="p">}</span>
<span class="n">metrics</span> <span class="p">{</span>
<span class="n">type</span> <span class="o">=></span> <span class="s2">"apache"</span>
<span class="n">meter</span> <span class="o">=></span> <span class="s2">"error.%{status}"</span>
<span class="n">add_tag</span> <span class="o">=></span> <span class="s2">"metric"</span>
<span class="n">ignore_older_than</span> <span class="o">=></span> <span class="mi">10</span>
<span class="p">}</span>
<span class="n">ruby</span> <span class="p">{</span>
<span class="n">tags</span> <span class="o">=></span> <span class="s2">"metric"</span>
<span class="c1"># code => "event.cancel if event['@fields']['error.504.rate_1m'] < 100"</span>
<span class="c1"># 2014/08/20: 每秒速率,所以要乘以60s。另,新版本没有了@fields,都存在顶级field里。</span>
<span class="n">code</span> <span class="o">=></span> <span class="s2">"event.cancel if event['error.504.rate_1m']*60 < 100"</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">output</span> <span class="p">{</span>
<span class="nb">exec</span> <span class="p">{</span>
<span class="n">tags</span> <span class="o">=></span> <span class="s2">"metric"</span>
<span class="n">command</span> <span class="o">=></span> <span class="s2">"sendsms.pl -m '%{error</span><span class="se">\.</span><span class="s2">504</span><span class="se">\.</span><span class="s2">rate_1m}'"</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>其中关键在两个 filter。 metrics 插件可以每5秒(前天刚更新了源码,这个值可以自己指定了)更新一次统计值,支持 <code class="highlighter-rouge">meter</code> 和 <code class="highlighter-rouge">timer</code> 两种,<code class="highlighter-rouge">timer</code> 除了 <code class="highlighter-rouge">count</code> 和 <code class="highlighter-rouge">rate_1|5|15m</code> 外,还可以统计 <code class="highlighter-rouge">min|max|stddev|mean</code> 和 <code class="highlighter-rouge">p1|5|10|90|95|99</code> 等详细数据。</p>
<p>ruby 插件则是直接 <code class="highlighter-rouge">eval</code> 写在 <code class="highlighter-rouge">code</code> 配置里的代码。</p>
<p>需要注意的是: <code class="highlighter-rouge">output</code> 里使用的时候,需要用 <code class="highlighter-rouge">\</code> 转义 <code class="highlighter-rouge">.</code>。否则配置解析后会认为变量不存在。这是目前官网文档上写的有问题的地方。我已經跟作者提过,或许过些天会修改。</p>
<p>值得一提的是:metrics 插件的输出是一个全新的 event,而不会去改变原先 grok 生成的 event。</p>
获取 Perl 程序中 GET 请求发向的具体 IP
2013-06-28T00:00:00+08:00
perl
http://chenlinux.com/2013/06/28/howto-get-peer-ipaddr-in-http-get
<p>在运维工作中我们经常需要检测用户访问是否正常,一般来说,直接通过 DNS 客户端获取 A 记录就可以满足需要。不过如果我们可以获得具体连接的 IP 地址,那么就可以缩小问题的判断范围,因为 DNS 的 A 记录通常是有多个的。</p>
<p>AE::HTTP 模块可以返回 sock 给用户进行具体操作,我们可以通过 sock 接口很简单的获得对端的 IP 地址:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nb">package</span> <span class="nn">Web::Checker::Util::</span><span class="nv">HTTP</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Moo</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">MooX::Types::MooseLike::</span><span class="nv">Base</span> <span class="sx">qw/Str Num/</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">AnyEvent::</span><span class="nv">HTTP</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">AnyEvent::</span><span class="nv">Socket</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">AnyEvent</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Time::</span><span class="nv">HiRes</span> <span class="sx">qw/time/</span><span class="p">;</span>
<span class="nv">has</span> <span class="nv">peer</span> <span class="o">=></span> <span class="p">(</span> <span class="nv">is</span> <span class="o">=></span> <span class="s">'rw'</span><span class="p">,</span> <span class="nv">isa</span> <span class="o">=></span> <span class="nv">Str</span> <span class="p">);</span>
<span class="nv">has</span> <span class="nv">reptime</span> <span class="o">=></span> <span class="p">(</span> <span class="nv">is</span> <span class="o">=></span> <span class="s">'rw'</span><span class="p">,</span> <span class="nv">isa</span> <span class="o">=></span> <span class="nv">Num</span> <span class="p">);</span>
<span class="nv">has</span> <span class="nv">clength</span> <span class="o">=></span> <span class="p">(</span> <span class="nv">is</span> <span class="o">=></span> <span class="s">'rw'</span><span class="p">,</span> <span class="nv">isa</span> <span class="o">=></span> <span class="nv">Num</span> <span class="p">);</span>
<span class="nv">has</span> <span class="nv">body</span> <span class="o">=></span> <span class="p">(</span> <span class="nv">is</span> <span class="o">=></span> <span class="s">'ro'</span><span class="p">,</span> <span class="nv">isa</span> <span class="o">=></span> <span class="nv">Str</span> <span class="p">);</span>
<span class="nv">has</span> <span class="nv">proxy</span> <span class="o">=></span> <span class="p">(</span> <span class="nv">is</span> <span class="o">=></span> <span class="s">'ro'</span><span class="p">,</span> <span class="nv">isa</span> <span class="o">=></span> <span class="nv">Str</span><span class="p">,</span> <span class="nv">default</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="nb">undef</span> <span class="p">}</span> <span class="p">);</span>
<span class="nv">has</span> <span class="nv">cv</span> <span class="o">=></span> <span class="p">(</span> <span class="nv">is</span> <span class="o">=></span> <span class="s">'ro'</span><span class="p">,</span> <span class="nv">default</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="nv">AnyEvent</span><span class="o">-></span><span class="nv">condvar</span> <span class="p">}</span> <span class="p">);</span>
<span class="k">sub </span><span class="nf">get</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$self</span><span class="p">,</span> <span class="nv">$url</span> <span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="nv">$self</span><span class="o">-></span><span class="nv">cv</span><span class="o">-></span><span class="nv">begin</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$begin</span> <span class="o">=</span> <span class="nb">time</span><span class="p">;</span>
<span class="nv">http_get</span> <span class="nv">$url</span><span class="p">,</span>
<span class="nv">proxy</span> <span class="o">=></span> <span class="nv">$self</span><span class="o">-></span><span class="nv">proxy</span><span class="p">,</span>
<span class="c1"># 就是这里发挥了作用,默认应该是直接返回 body 字符串的</span>
<span class="nv">want_body_handle</span> <span class="o">=></span> <span class="mi">1</span><span class="p">,</span>
<span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$hdl</span><span class="p">,</span> <span class="nv">$headers</span> <span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$port</span><span class="p">,</span> <span class="nv">$peer</span> <span class="p">)</span> <span class="o">=</span>
<span class="nn">AnyEvent::Socket::</span><span class="nv">unpack_sockaddr</span> <span class="nb">getpeername</span> <span class="nv">$hdl</span><span class="o">-></span><span class="p">{</span><span class="nv">fh</span><span class="p">};</span>
<span class="nv">$self</span><span class="o">-></span><span class="nv">peer</span><span class="p">(</span> <span class="nn">AnyEvent::Socket::</span><span class="nv">format_address</span> <span class="nv">$peer</span> <span class="p">);</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$headers</span><span class="o">-></span><span class="p">{</span><span class="nv">Status</span><span class="p">}</span> <span class="o">=~</span> <span class="sr">/^2/</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$end</span> <span class="o">=</span> <span class="nb">time</span><span class="p">;</span>
<span class="nv">$self</span><span class="o">-></span><span class="nv">reptime</span><span class="p">(</span> <span class="nv">$end</span> <span class="o">-</span> <span class="nv">$begin</span> <span class="p">);</span>
<span class="nv">$self</span><span class="o">-></span><span class="nv">clength</span><span class="p">(</span> <span class="nv">$headers</span><span class="o">-></span><span class="p">{</span><span class="s">'content-length'</span><span class="p">}</span> <span class="p">);</span>
<span class="nv">$self</span><span class="o">-></span><span class="nv">cv</span><span class="o">-></span><span class="nv">end</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="nv">$self</span><span class="o">-></span><span class="nv">cv</span><span class="o">-></span><span class="nb">recv</span><span class="p">;</span>
<span class="p">}</span>
<span class="mi">1</span><span class="p">;</span>
</code></pre>
</div>
<p>其实 AE::HTTP 还可以在 <code class="highlighter-rouge">tcp_connect</code> 的时候获取 sock,这时候就需要自己用 <code class="highlighter-rouge">AnyEvent::Handle</code> 写一遍 <code class="highlighter-rouge">AnyEvent::HTTP::tcp_connect</code> 已经写过的东西了(当然如果你本来就打算干点别的事情,那就是另外一回事情了)~~</p>
计算两个时间点之间隔了几天
2013-06-24T00:00:00+08:00
perl
http://chenlinux.com/2013/06/24/who-to-calculate-days-between-two-date
<p>两个时间点字符串,像这样:<code class="highlighter-rouge">2013-06-21</code>,怎么计算相距多少天呢?</p>
<p>有两种办法。</p>
<h2 id="datetime-">DateTime 模块</h2>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nv">DateTime</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">List::</span><span class="nv">MoreUtils</span> <span class="sx">qw(zip)</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Data::</span><span class="nv">Dumper</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">Dumper</span><span class="p">(</span>
<span class="nv">DateTime</span><span class="o">-></span><span class="k">new</span><span class="p">(</span> <span class="nv">zip</span> <span class="nv">@</span><span class="p">{</span> <span class="p">[</span><span class="sx">qw/year month day/</span><span class="p">]</span> <span class="p">},</span>
<span class="nv">@</span><span class="p">{</span> <span class="p">[</span> <span class="nb">split</span> <span class="sr">/-/</span><span class="p">,</span> <span class="s">'2013-06-21'</span> <span class="p">]</span> <span class="p">}</span> <span class="p">)</span><span class="o">-></span><span class="nv">subtract_datetime</span><span class="p">(</span>
<span class="nv">DateTime</span><span class="o">-></span><span class="k">new</span><span class="p">(</span>
<span class="nv">zip</span> <span class="nv">@</span><span class="p">{</span> <span class="p">[</span><span class="sx">qw/year month day/</span><span class="p">]</span> <span class="p">},</span>
<span class="nv">@</span><span class="p">{</span> <span class="p">[</span> <span class="nb">split</span> <span class="sr">/-/</span><span class="p">,</span> <span class="s">'2012-05-20'</span> <span class="p">]</span> <span class="p">}</span>
<span class="p">)</span>
<span class="p">)</span><span class="o">-></span><span class="nv">deltas</span>
<span class="p">);</span>
</code></pre>
</div>
<p>缺点是 <code class="highlighter-rouge">DateTime::Duration</code> 的 <code class="highlighter-rouge">days()</code> 只能返回进位 <code class="highlighter-rouge">months()</code> 之后剩余的天数。所以这里只能输出整个 <code class="highlighter-rouge">deltas()</code> 来看。</p>
<h2 id="timestamp-">timestamp 时间戳</h2>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nv">POSIX</span> <span class="sx">qw(mktime)</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">trans</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">@str</span> <span class="o">=</span> <span class="nb">split</span> <span class="sr">/-/</span><span class="p">,</span> <span class="nb">shift</span><span class="p">;</span>
<span class="nv">mktime</span><span class="p">(</span>
<span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="nv">$str</span><span class="p">[</span><span class="mi">2</span><span class="p">],</span>
<span class="nv">$str</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">-</span> <span class="mi">1</span><span class="p">,</span>
<span class="nv">$str</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">-</span> <span class="mi">1900</span><span class="p">,</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="k">my</span> <span class="nv">$dt1</span> <span class="o">=</span> <span class="nv">trans</span><span class="p">(</span><span class="s">'1999-05-21'</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$dt2</span> <span class="o">=</span> <span class="nv">trans</span><span class="p">(</span><span class="s">'2013-06-26'</span><span class="p">);</span>
<span class="k">print</span><span class="p">(</span> <span class="p">(</span> <span class="nv">$dt2</span> <span class="o">-</span> <span class="nv">$dt1</span> <span class="p">)</span> <span class="o">/</span> <span class="p">(</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">24</span> <span class="p">)</span> <span class="p">);</span>
</code></pre>
</div>
<p>这里就是要注意,<code class="highlighter-rouge">mktime</code> 里的 <code class="highlighter-rouge">month</code> 是以 0 开始的,<code class="highlighter-rouge">year</code> 是从 1900 开始的。</p>
<hr />
<p><strong>2014 年 01 月 22 日更新:</strong></p>
<p>在2013 年底的 advent calendar 和 perlmaven 上学习到了另外两个模块,这里补充一下:</p>
<h2 id="timepiece-">Time::Piece 模块</h2>
<p>这个模块是 Perl5 的corelist 模块,所以不用另外安装就能使用:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nn">Time::</span><span class="nv">Piece</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$t1</span> <span class="o">=</span> <span class="nn">Time::</span><span class="nv">Piece</span><span class="o">-></span><span class="nv">strptime</span><span class="p">(</span><span class="s">'2013-06-26'</span><span class="p">,</span> <span class="s">'%Y-%m-%d'</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$t2</span> <span class="o">=</span> <span class="nn">Time::</span><span class="nv">Piece</span><span class="o">-></span><span class="nv">strptime</span><span class="p">(</span><span class="s">'2012-06-21 GMT'</span><span class="p">,</span> <span class="s">'%Y-%m-%d %Z'</span><span class="p">);</span>
<span class="k">print</span> <span class="o">+</span><span class="p">(</span><span class="nv">$t1</span> <span class="o">-</span> <span class="nv">$t2</span><span class="p">)</span><span class="o">-></span><span class="nv">days</span><span class="p">;</span>
</code></pre>
</div>
<p>Time::Piece 模块重载了加减号,所以直接两个时间相减后就得到了 Time::Seconds 对象,然后调用 <code class="highlighter-rouge">days</code> 方法返回具体天数就可以了。</p>
<p>这里有个奇怪的问题,在采用 <code class="highlighter-rouge">strptime</code> 方法解析创建对象的时候,<code class="highlighter-rouge">%Z</code> 格式似乎除了 <code class="highlighter-rouge">GMT</code> 之外写其他的都会爆出:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Error parsing time at /usr/lib/perl/5.14/Time/Piece.pm line 469.
</code></pre>
</div>
<p>这个真的很诡异了。</p>
<p><strong>2014 年 01 月 23 日补充:</strong></p>
<p>去看了一下 <a href="https://github.com/rjbs/Time-Piece/blob/master/Piece.xs">Piece.xs</a> 的内容,发现虽然文档上说是学习的 <a href="http://www.opensource.apple.com/source/libc/libc-583/stdtime/strptime-fbsd.c">FreeBSD 的 strptime</a> 实现,但是差的也太多了~直接里面 <code class="highlighter-rouge">_strptime</code> 函数关于时区的就一个 <code class="highlighter-rouge">*got_GMT</code> 真假判断 ==!</p>
<p>完整的 strptime 见 <a href="https://metacpan.org/pod/POSIX::strptime">POSIX::strptime</a> 模块,或许我可以写一个扩展?</p>
<h2 id="datetimemoonpig-">DateTime::Moonpig 模块</h2>
<p>这个模块是最近出的,属于 DateTime 模块的接口封装和优化。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nn">DateTime::</span><span class="nv">Moonpig</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$t3</span> <span class="o">=</span> <span class="nn">DateTime::</span><span class="nv">Moonpig</span><span class="o">-></span><span class="k">new</span><span class="p">(</span><span class="nv">year</span> <span class="o">=></span> <span class="mi">2013</span><span class="p">,</span> <span class="nv">month</span> <span class="o">=></span> <span class="mi">6</span><span class="p">,</span> <span class="nv">day</span> <span class="o">=></span> <span class="mi">26</span><span class="p">,</span> <span class="nv">time_zone</span> <span class="o">=></span> <span class="s">'America/New_York'</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$t4</span> <span class="o">=</span> <span class="nn">DateTime::</span><span class="nv">Moonpig</span><span class="o">-></span><span class="k">new</span><span class="p">(</span><span class="nv">year</span> <span class="o">=></span> <span class="mi">2012</span><span class="p">,</span> <span class="nv">month</span> <span class="o">=></span> <span class="mi">6</span><span class="p">,</span> <span class="nv">day</span> <span class="o">=></span> <span class="mi">21</span><span class="p">,</span> <span class="nv">time_zone</span> <span class="o">=></span> <span class="s">'GMT'</span><span class="p">);</span>
<span class="k">print</span> <span class="nb">int</span><span class="p">(</span> <span class="p">(</span><span class="nv">$t3</span> <span class="o">-</span> <span class="nv">$t4</span><span class="p">)</span> <span class="o">/</span> <span class="p">(</span><span class="mi">60</span> <span class="o">*</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">24</span><span class="p">)</span> <span class="p">);</span>
</code></pre>
</div>
<p>从示例可以看出两点优化:</p>
<ol>
<li>可以灵活调整 DateTime::Moonpig 对象的时区,而不用分别 <code class="highlighter-rouge">use DateTime;use DateTime::TimeZone</code>;</li>
<li>直接加减返回的不再是那个不好用的 <code class="highlighter-rouge">DateTime::Duration</code> 对象而是秒数。</li>
</ol>
如何去除 rpmbuild 自动发现的依赖关系
2013-06-21T00:00:00+08:00
linux
perl
bash
http://chenlinux.com/2013/06/21/remove-auto-deps-from-rpmbuild
<p>同事在用简单的 SPEC 配置打包 nagios 套件的时候,发现最后生成的 RPM 包附加了很多依赖关系。其中 <code class="highlighter-rouge">perl-Net-SNMP</code> 这个包,是服务器默认安装中没有的。这也不是什么大问题。不过这个出现还是蛮奇怪的。值得研究一下。</p>
<p>后来在 <code class="highlighter-rouge">/usr/lib/rpm/</code> 目录下发现了一系列脚本,诸如<code class="highlighter-rouge">javadeps</code>/<code class="highlighter-rouge">perl.req</code>/<code class="highlighter-rouge">pythondeps</code>/<code class="highlighter-rouge">find-requires</code>/<code class="highlighter-rouge">mono-find-requires</code>等等。</p>
<p>这些脚本的作用是,用 <code class="highlighter-rouge">file</code> 命令判断文件,如果是二进制的,用ldd判断依赖;如果是脚本,过滤文件中对应的 <code class="highlighter-rouge">use</code>/<code class="highlighter-rouge">requires</code>/<code class="highlighter-rouge">import</code> 语句。这样就可以找出来源代码的内部依赖了。</p>
<p>那么怎么才能跳过这段逻辑呢?</p>
<p>最暴力的办法,这些文件都是 bash 或者 perl 脚本,直接修改。</p>
<p>但是还可以文明一点,像下面这段,添加在 SPEC 文件中:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> %setup
%prep
cat <span class="sh"><< \EOF > %{name}-req
#!/bin/sh
%{__perl_requires} $* |\
sed -e '/perl(Net::SNMP)/d'
EOF
</span> %define __perl_requires %<span class="o">{</span>_builddir<span class="o">}</span>/%<span class="o">{</span>name<span class="o">}</span>-%<span class="o">{</span>version<span class="o">}</span>/%<span class="o">{</span>name<span class="o">}</span>-req
chmod 755 %<span class="o">{</span>__perl_requires<span class="o">}</span>
</code></pre>
</div>
<p>这里重定义了一个脚本,原先的定义在 <code class="highlighter-rouge">/usr/lib/rpm/macros</code> 中,是:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="c">#%__find_provides /usr/lib/rpm/rpmdeps --provides</span>
<span class="c">#%__find_requires /usr/lib/rpm/rpmdeps --requires</span>
%__find_provides /usr/lib/rpm/find-provides
%__find_requires /usr/lib/rpm/find-requires
<span class="c">#%__perl_provides /usr/lib/rpm/perldeps.pl --provides</span>
<span class="c">#%__perl_requires /usr/lib/rpm/perldeps.pl --requires</span>
%__perl_provides /usr/lib/rpm/perl.prov
%__perl_requires /usr/lib/rpm/perl.req
%__python_provides /usr/lib/rpm/pythondeps.sh --provides
%__python_requires /usr/lib/rpm/pythondeps.sh --requires
%__mono_provides /usr/lib/rpm/mono-find-provides %<span class="o">{</span>_builddir<span class="o">}</span>/%<span class="o">{</span>?buildsubdir<span class="o">}</span> %<span class="o">{</span>buildroot<span class="o">}</span> %<span class="o">{</span>_libdir<span class="o">}</span>
%__mono_requires /usr/lib/rpm/mono-find-requires %<span class="o">{</span>_builddir<span class="o">}</span>/%<span class="o">{</span>?buildsubdir<span class="o">}</span> %<span class="o">{</span>buildroot<span class="o">}</span> %<span class="o">{</span>_libdir<span class="o">}</span>
</code></pre>
</div>
<p>然后将加入了 <code class="highlighter-rouge">sed</code> 命令的新脚本定位为新的 MACROS 变量给 <code class="highlighter-rouge">rpmbuild</code> 后续使用。</p>
通过 Rex 命令行参数向动态服务器组发起任务
2013-06-20T00:00:00+08:00
devops
rex
sqlite
perl
http://chenlinux.com/2013/06/20/rex-task-params-for-dynamic-server-group
<p>Rex 默认的服务器组定义方式有三种,直接写在 <code class="highlighter-rouge">Rexfile</code> 文件中;每行一个写成 IP 列表保存成文件,然后通过 <code class="highlighter-rouge">lookup_file</code> 读取;把组名和 IP 写成 <code class="highlighter-rouge">.ini</code> 格式文件,通过 <code class="highlighter-rouge">groups_file "$name.ini"</code> 一次性获取。</p>
<p>如果服务器信息存在数据库里,那么可以通过 <code class="highlighter-rouge">Rex::Commands::DB</code> 来快速读取数据库信息,构建服务器组。不过,如果我们是想从数据库中根据查询条件,动态获取服务器列表完成指定任务的话,就没法提前定义好 <code class="highlighter-rouge">group</code> 了。这个时候,怎么办呢?</p>
<p>我们可以利用 <code class="highlighter-rouge">task</code> 可以接受命令行参数这个特点,完成这个功能:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nn">Rex::Commands::</span><span class="nv">DB</span> <span class="p">{</span>
<span class="nv">dsn</span> <span class="o">=></span> <span class="s">"dbi:SQLite:dbname=/etc/puppet/webui/node.db"</span><span class="p">,</span>
<span class="nv">user</span> <span class="o">=></span> <span class="s">""</span><span class="p">,</span>
<span class="nv">password</span> <span class="o">=></span> <span class="s">""</span><span class="p">,</span>
<span class="p">};</span>
<span class="nv">task</span> <span class="s">"sqlite"</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$param</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$role</span> <span class="o">=</span> <span class="nv">$param</span><span class="o">-></span><span class="p">{</span><span class="nv">role</span><span class="p">};</span>
<span class="k">my</span> <span class="nv">$class</span> <span class="o">=</span> <span class="nv">$param</span><span class="o">-></span><span class="p">{</span><span class="nv">class</span><span class="p">};</span>
<span class="k">my</span> <span class="nv">$todo</span> <span class="o">=</span> <span class="nv">$param</span><span class="o">-></span><span class="p">{</span><span class="nv">todo</span><span class="p">};</span>
<span class="nb">grep</span> <span class="p">{</span> <span class="nv">run_task</span> <span class="nv">$todo</span><span class="p">,</span> <span class="nv">on</span> <span class="o">=></span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="nv">ip</span><span class="p">}</span> <span class="p">}</span> <span class="nv">db</span> <span class="nb">select</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">fields</span> <span class="o">=></span> <span class="s">"ip"</span><span class="p">,</span>
<span class="nv">from</span> <span class="o">=></span> <span class="s">"node_info"</span><span class="p">,</span>
<span class="nv">where</span> <span class="o">=></span> <span class="s">"role like '$role\%' and classes like '\%${class}\%'"</span><span class="p">,</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="nv">task</span> <span class="s">'hello'</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">say</span> <span class="nv">run</span> <span class="s">"w"</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<p>然后这样运行命令即可:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>rex sqlite --role<span class="o">=</span>cdn --class<span class="o">=</span>nginx --todo<span class="o">=</span>hello
</code></pre>
</div>
【Etsy 的 Kale 系统】skyline 的过滤算法
2013-06-19T00:00:00+08:00
monitor
python
http://chenlinux.com/2013/06/19/skyline-algorithms-intro
<p>监控大户 Etsy 最近有公布了一个全新的监控分析系统,叫 Kale,博客地址:<a href="http://codeascraft.com/2013/06/11/introducing-kale/">http://codeascraft.com/2013/06/11/introducing-kale/</a>。</p>
<p>上一篇博客介绍了安装部署和数据导入的方法。但是对 <code class="highlighter-rouge">skyline</code> 组件的过滤原理没有做研究。今天花了点时间看 wiki 和源码,大概搞清楚了 <code class="highlighter-rouge">skyline</code> 的工作方式。很有趣,值得记录一下。</p>
<p>同样作为时间序列存储的 <code class="highlighter-rouge">rrdtool</code> 和 <code class="highlighter-rouge">graphite</code>,都偏重在预测算法,也就是说根据现有数据推测下一个数据应该是多少;而 <code class="highlighter-rouge">skyline</code> 则是根据现有数据统计最新数据是否异常。</p>
<p>目前,<code class="highlighter-rouge">skyline</code> 一共提供了 7 个异常检测算法,如果有 5 个以上认为是异常,那么 <code class="highlighter-rouge">skyline</code> 就认为这个序列异常了。(当然,这都是可以修改的)</p>
<p>异常检测算法实际写在了 <code class="highlighter-rouge">src/analyzer/algorithms.py</code> 里,包括有:</p>
<h3 id="firsthouraverage">first_hour_average</h3>
<p>这是最简单的。先求本周期内最前面的第一个小时的平均值和标准差,然后和最新的三个值的平均值(<code class="highlighter-rouge">tail_avg()</code>,这是后面多数算法都通用的做法)做比较。如果 <code class="highlighter-rouge">tail_avg</code> 和 第一小时平均值的差距大于 3 倍的标准差,那么认定为异常。</p>
<h3 id="simplestddevfrommovingaverage">simple_stddev_from_moving_average</h3>
<p>把上面算法的范围扩大化,求的是整个周期内全部数据的平均值和标准差。</p>
<h3 id="stddevfrommovingaverage">stddev_from_moving_average</h3>
<p>在上面算法的基础上,采用指数加权移动平均值。对周期内采点数量较少的情况更好一些。</p>
<h3 id="meansubtractioncumulation">mean_subtraction_cumulation</h3>
<p>做法是这样的:</p>
<ol>
<li>排除最后一个值;</li>
<li>求剩余序列的平均值;</li>
<li>全序列减去上面这个平均值;</li>
<li>求剩余序列的标准差;</li>
<li>判断全序列最后一个值是否大于 3 倍的标准差</li>
</ol>
<p>在代码中本来还计算了一次序列的指数加权移动平均值,但是算完了却没用,感觉怪怪的。</p>
<h3 id="leastsquares">least_squares</h3>
<p>采用最小二乘法拟近时间序列,然后用实际值减去拟近值得到新序列。然后判断新序列的最后三个值的平均值是否大于 3 倍的新序列标准差。</p>
<p>所谓最小二乘法,简单说就是对一个 [x, y] 序列,会有一对常数 [m, c],让 <code class="highlighter-rouge">Y = mx + c</code> 等式中的 Y 和 y 在全序列上最接近。</p>
<h3 id="histogrambins">histogram_bins</h3>
<p>将整个周期序列的数据按照直方图统计法归入 15 个直方中,然后看最后三个值的平均值属于这 15 个直方的具体哪个。如果这个直方中包含的数据小于 20 个,判断为异常。</p>
<p>从算法中可以知道,如果周期内数据量不够,很容易被判断为异常的。</p>
<h3 id="grubbs">grubbs</h3>
<p>将整个周期序列的数据按照格拉布斯法求异常值。</p>
<p>标准的格拉布斯法是这样的:</p>
<ol>
<li>从小到大排序;</li>
<li>求序列的平均值和标准差;</li>
<li>计算最小值和最大值与平均值的差距,更大的那个为可疑值;</li>
<li>可疑值减去平均值,再除以标准差,如果大于格拉布斯临界值,那么就是异常值;</li>
<li>排除异常值,对剩余序列循环做 1-5 步骤。</li>
</ol>
<p>这里只用判断时间序列的最后是否异常,所以直接将最后三个值的平均值作为可疑值判断是否异常即可。</p>
<p><strong>2013 年 07 月 23 日更新</strong></p>
<p>新增了一个异常算法,现在有 8 个了,要通过 6 个才算真异常。</p>
<p>新增的是”绝对中值偏差法”</p>
<h3 id="medianabsolutedeviation">median_absolute_deviation</h3>
<p>具体实现是:序列的最后一个值,比该序列的绝对中值大 6 倍以上,即判断为异常。</p>
<p>注意这里是中值,不是平均值。</p>
<p><strong>2013 年 08 月 14 日更新</strong></p>
<p>新增一个异常算法,现在有 9 个了。</p>
<p>新增的是”柯尔莫诺夫-斯米尔诺夫检验法”</p>
<h3 id="kolmogorov-smirnovtest">Kolmogorov-Smirnov_test</h3>
<p>具体实现是:计算序列内最近十分钟的数值的ks测试分布,然后计算序列中最近一个小时前到十分钟前这 50 分钟的数值的ks测试分布;如果两个分布相差较大,即判断为异常。</p>
【Etsy 的 Kale 系统】简介、部署和应用
2013-06-18T00:00:00+08:00
monitor
python
perl
ruby
elasticsearch
redis
graphite
http://chenlinux.com/2013/06/18/etsy-kale-intro
<p>监控大户 Etsy 最近有公布了一个全新的监控分析系统,叫 Kale,博客地址:<a href="http://codeascraft.com/2013/06/11/introducing-kale/">http://codeascraft.com/2013/06/11/introducing-kale/</a>。</p>
<p>目前的介绍内容比较简单。两个组件 <code class="highlighter-rouge">skyline</code> 和 <code class="highlighter-rouge">oculus</code> 之间的关系也还没搞清楚。大概上, <code class="highlighter-rouge">skyline</code> 是一个 python 程序,接受 <code class="highlighter-rouge">cPickle</code> 和 <code class="highlighter-rouge">MessagePack</code> 两种数据包,解压后的数据格式类似 <code class="highlighter-rouge">graphite</code> 接收的,然后存在 <code class="highlighter-rouge">Redis-server</code> 中。在 webapp 上提供一个类似 <code class="highlighter-rouge">rrdtool</code> 的功能,显示触发阈值线的趋势图(不触发的不会显示,自动过滤了)。</p>
<p>安装步骤:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> pip install -r requirements.txt
apt-get install -y numpy scipy
pip install pandas patsy statsmodels msgpack_python
cp src/settings.py.example src/settings.py
mkdir /var/log/skyline
mkdir /var/run/skyline
mkdir /var/log/redis
<span class="c"># 必须用最新版的 redis-server 才能正常存储</span>
wget http://redis.googlecode.com/files/redis-2.6.13.tar.gz
tar zxvf redis-2.6.13.tar.gz
<span class="nb">cd </span>redis-2.6.13
make
./src/redis-server ../bin/redis.conf
<span class="nb">cd</span> ../src
<span class="c"># 这里会启动 UDP 2024 端口接受 cpickle 包,2025 端口接受 msgpack 包</span>
../bin/horizon.d start
../bin/analyzer.d start
<span class="c"># 这里会启动 TCP 1500 端口接受 web 访问</span>
../bin/webapp.d start
<span class="c"># 测试是否正常</span>
<span class="nb">cd</span> ../utils
./seed_data.py
</code></pre>
</div>
<p><code class="highlighter-rouge">oculus</code> 是一个 rack 应用,需要定时从 <code class="highlighter-rouge">skyline</code> 中导入数据到 <code class="highlighter-rouge">ElasticSearch</code> 中。同时,<code class="highlighter-rouge">oculus</code> 还提供了一个 <code class="highlighter-rouge">ElasticSearch</code> 分析器插件,可以在 ES 中完成 <code class="highlighter-rouge">FastDTW</code> 和 <code class="highlighter-rouge">Euclidian</code> 两种位移算法(用来给不同时间序列的近似度打分)。在rack 页面上,提供搜索框,你可以提交一个 metric 名称——经过测试,目前应该是采用完全匹配的方式搜索——然后展示这个 metric 的图形,以及按照 score 打分排序的近似时间序列。</p>
<ul>
<li>欧几里德算法原理:根据两点的坐标系计算直线距离;</li>
<li>动态时间归整原理:将时间序列进行延伸或者缩短,然后再计算。</li>
</ul>
<p><a href="http://www.cnblogs.com/kemaswill/archive/2013/04/18/3028610.html">http://www.cnblogs.com/kemaswill/archive/2013/04/18/3028610.html</a></p>
<p>安装步骤:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="c"># 只能用 0.20.5 版,0.90 版目前不支持</span>
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.5.tar.gz
tar zxvf elasticsearch-0.20.5.tar.gz
mv elasticsearch-0.20.5 /opt/elasticsearch
<span class="c"># 编译插件</span>
cp -r resources/elasticsearch-oculus-plugin /opt/elasticsearch/
<span class="nb">pushd</span> /opt/elasticsearch/elasticsearch-oculus-plugin
rake build
cp OculusPlugins.jar /opt/elasticsearch/lib/OculusPlugins.jar
<span class="c"># 加载分析器和脚本</span>
cat >>/opt/elasticsearch/config/elasticsearch.yml<span class="sh"><<EOF
script.native:
oculus_euclidian.type: com.etsy.oculus.tsscorers.EuclidianScriptFactory
oculus_dtw.type: com.etsy.oculus.tsscorers.DTWScriptFactory
EOF
</span> <span class="c"># 启动</span>
/opt/elasticsearch/bin/elasticsearch
<span class="nb">popd
</span>bundle install
mkdir /var/run/oculus
mkdir /var/log/oculus
<span class="c"># 启动 worker 进程,这是import.rb 和 ES 交流的渠道</span>
rake resque:start_workers
<span class="c"># 编辑 config/config.yml,注意里面ES一定要提供两台,哪怕写一个127.0.0.1一个localhost,后面 import 会验证数目</span>
vi config/config.yml
<span class="c"># 从 skyline 导入数据</span>
./scripts/import.rb
<span class="nb">echo</span> <span class="s1">'*/2 * * * * ~/oculus/scripts/import.rb &> /var/log/oculus/import.log'</span> >> cron.list
crontab -u root cron.list
<span class="c"># 启动web</span>
thin start
<span class="c"># 默认用户密码都是admin,需要先点击初始化</span>
gnome-open localhost:3000/admin
</code></pre>
</div>
<p><code class="highlighter-rouge">oculus</code> 的测试我是做出来了。如图:</p>
<p><img src="/images/uploads/oculus.png" alt="oculus" /></p>
<p>这个数据我是通过 perl 生成的随机数,所以也没什么近似队列了。展示一下脚本,这样说明我们可以通过其他脚本扩展 Kale 系统的用途。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="c1">#!/usr/bin/env perl</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Data::</span><span class="nv">MessagePack</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">AnyEvent::Handle::</span><span class="nv">UDP</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$mp</span> <span class="o">=</span> <span class="nn">Data::</span><span class="nv">MessagePack</span><span class="o">-></span><span class="k">new</span><span class="o">-></span><span class="nv">utf8</span><span class="o">-></span><span class="nv">prefer_integer</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$cv</span> <span class="o">=</span> <span class="nv">AnyEvent</span><span class="o">-></span><span class="nv">condvar</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$sock</span> <span class="o">=</span> <span class="nn">AnyEvent::Handle::</span><span class="nv">UDP</span><span class="o">-></span><span class="k">new</span><span class="p">(</span>
<span class="nb">connect</span> <span class="o">=></span> <span class="p">[</span> <span class="s">'127.0.0.1'</span><span class="p">,</span> <span class="s">'2025'</span> <span class="p">],</span>
<span class="nv">on_recv</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="p">},</span>
<span class="nv">autoflush</span> <span class="o">=></span> <span class="mi">1</span><span class="p">,</span>
<span class="p">);</span>
<span class="k">my</span> <span class="nv">$timer</span> <span class="o">=</span> <span class="nv">AnyEvent</span><span class="o">-></span><span class="nv">timer</span><span class="p">(</span>
<span class="nv">after</span> <span class="o">=></span> <span class="mi">0</span><span class="p">,</span>
<span class="nv">interval</span> <span class="o">=></span> <span class="mi">5</span><span class="p">,</span>
<span class="nv">cb</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">print</span> <span class="s">"send...\n"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$data</span> <span class="o">=</span> <span class="p">[</span> <span class="s">'localhost.loadavg'</span><span class="p">,</span> <span class="p">[</span> <span class="nb">time</span><span class="p">(),</span> <span class="nb">rand</span><span class="p">()</span> <span class="o">*</span> <span class="mi">2</span> <span class="p">]</span> <span class="p">];</span>
<span class="k">my</span> <span class="nv">$packed</span> <span class="o">=</span> <span class="nv">$mp</span><span class="o">-></span><span class="nb">pack</span><span class="p">(</span><span class="nv">$data</span><span class="p">);</span>
<span class="nv">$sock</span><span class="o">-></span><span class="nv">push_send</span><span class="p">(</span><span class="s">"$packed"</span><span class="p">);</span>
<span class="p">},</span>
<span class="p">);</span>
<span class="nv">$cv</span><span class="o">-></span><span class="nb">recv</span><span class="p">;</span>
</code></pre>
</div>
<p>从源码中,看到还有 <code class="highlighter-rouge">ganglia_to_skyline.rb</code> 脚本。目前看,<code class="highlighter-rouge">Kale</code> 应该是想着用 <code class="highlighter-rouge">skyline</code> 代替 <code class="highlighter-rouge">graphite-web</code>,得用 <code class="highlighter-rouge">redis</code> 来代替 <code class="highlighter-rouge">graphite-whisper</code>,不过我觉得似乎意义不是很大,还不如直接把数据存入 <code class="highlighter-rouge">ElasticSearch</code>,形成一套类似 <code class="highlighter-rouge">openTSDB</code> 的,但是完全基于 ES 的高扩展分布式方案。</p>
【翻译】运维的85条规则
2013-06-12T00:00:00+08:00
http://chenlinux.com/2013/06/12/translate-85-operational-rules
<p>2007 年,时任虚拟世界游戏公司 Vivaty 运维副总裁的 Jon Prall 在他的个人博客上发表过一篇<a href="http://jprall.typepad.com/blog/2010/10/85-operational-rules.html">《运维的85条规则》</a>。2010 年他跳槽到视频电话公司 Tango 之初,做了两处更新,兹翻译如下:</p>
<ol>
<li>
<p>容量第一,优化第二——这条规则在故障发生时生效。在宕机的时候别研究什么优化,先恢复设备。</p>
</li>
<li>
<p>保留所有可以捕获的记录——以 PostgresQL 为例,包括有 WAL 文件,Slony 复制,快照技术,基于硬盘的 DB 版本(快照附带的)</p>
</li>
<li>
<p>不要因为优化引入更多问题。通常我们解决问题时做出来的东西都会转变成之后运维工作的负担。请确认为运维工作开发的那些工具已经完全交付使用。这些东西经常无法正常运行结果要返回开发组重来。更重要的,这种变更请求通常会打破团队原本安排好的工作计划。</p>
</li>
<li>
<p>保持简单,不要让事情变得太复杂,聪明的你一定可以做到的。</p>
</li>
<li>
<p>谨慎使用缓存以保护那些难以水平扩展的资源。当然,如果你可以水平扩展它,那么给他加缓存层就不用考虑太多。一旦用上了缓存层,它的目的应该是提高最终用户的访问性能,而不是增加网站的容量。否则,你不过是给自己加上了一个新的非常不可靠的瓶颈。他们潜在的负面影响可能危及整个系统。事实上缓存层失效带来的,经常是雪崩式的级联故障。</p>
</li>
<li>
<p>不要什么都自己写代码实现,也不要什么都从厂家买——要在适当的时候采用适当的工具。</p>
</li>
<li>
<p>谈判——和真正有实力的厂家谈判的唯一办法就是提前做好功课,准备好一切可行项。这样一旦有必要,你可以从你的首选厂家里选择离开。不用搞虚张声势那套了。</p>
</li>
<li>
<p>永远要准备好 N+1 的服务器。如果 N 等于 1,那么不管什么情况都不要动用这个 +1 的设备,专职等待 N 失效后的接管。当你使用冗余的服务器来均衡负载的时候,就只有49%或者更少的容量可管理了。通常我们会获得 N+2 的机会——一定要好好利用起来。</p>
</li>
<li>
<p>数据丢失是任何一家公司都不敢冒的风险——这是一条普遍真理。丢失数据造成的损耗远远超过用于保证数据不丢失的花费。</p>
</li>
<li>
<p>随时随地的并行化——这是一种很重要的思维方式。比如,如果 MogileFS 设置为位置感知的方式并且需要实时复制,那么每个 MogileFS 服务器都必须可以复制自己的数据到负载均衡器指定的另一端。只要有可能,尽量实现这种多对多的方式。</p>
</li>
<li>
<p>RTFM——就在今天我还要阅读一对 RAID 卡的说明书来比较他们微妙的差异。魔鬼在于细节。像做家庭作业一样读文档吧!</p>
</li>
<li>
<p>了解每一层上的瓶颈以及如何发现瓶颈。必须要知道你是在磁盘,内存,还是 CPU 上受限制了,搞清楚这个其实挺简单的。</p>
</li>
<li>
<p>要有一个固定的容量管理流程——而且是主动式的,不是被动式的。要知道系统的弱点在哪里,让实际负荷曲线跑到容量曲线之上是极度危险的。</p>
</li>
<li>
<p>不促成失败,也不惧怕改变。</p>
</li>
<li>
<p>不要吸进你自己的废气。别以为你现在的工作结果会变成未来你如何工作的动力。</p>
</li>
<li>
<p>运维人员要写的代码是运维工具,而不是应用软件。</p>
</li>
<li>
<p>不要低估运维团队中项目经理、技术作者、金融分析师的价值。这些人通常比你给的工资值钱多了。</p>
</li>
<li>
<p>监控所有的东西——报警只用在异动的时候,其他的都记录下来供趋势分析。</p>
</li>
<li>
<p>要有一个固定的流程来查看每个地方的趋势数据。</p>
</li>
<li>
<p>不要让监控太吵闹,那样很快就变得没作用了。</p>
</li>
<li>
<p>确保你的监控系统简单易用到公司里每个人都能上手。监控数据指标转换成为业务指标、市场指标和销售指标等等的频率可能高的让你吃惊。</p>
</li>
<li>
<p>只在可以做出相应改变的地方做总结,否则就是白白浪费时间。</p>
</li>
<li>
<p>总结要公开,同时附上事件相关的数据。这样大家可以很容易的找到总结的关键点并且跳转到对应数据。</p>
</li>
<li>
<p>要让技术的每一个点都有人员在负责。</p>
</li>
<li>
<p>同时为这些负责人准备好备份人员。</p>
</li>
<li>
<p>不断发招聘——哪怕没有名额了。</p>
</li>
<li>
<p>做自己最严厉的批评者。不管自己或者自认多聪明,总有可以提高的地方。</p>
</li>
<li>
<p>多往外看,拿自身的水平和尽量多的公司的职位需求做对比。</p>
</li>
<li>
<p>每年参加一个技术交流大会。如果一年有好几个,那选最好的那一个去就够了。</p>
</li>
<li>
<p>买你需要的而不是你想要的。绝不摘下你公司的帽子换上那个写着“对我来说什么最简单最安全”的。</p>
</li>
<li>
<p>只做对业务最好的事情,哪怕这件事是让你滚蛋……</p>
</li>
<li>
<p>问责制度正规化——记录承诺,事后追究没有完成者。</p>
</li>
<li>
<p>不允许重复失败。听起来有些过于苛责了。不过要区分不可挽回的失误和失误的差别。</p>
</li>
<li>
<p>无情——因为对手都是无情的。</p>
</li>
<li>
<p>工作是你要在完成的时候亲自署名的东西。署名同时也意味着完成任务。</p>
</li>
<li>
<p>保持对外的可用联络。</p>
</li>
<li>
<p>创业的伙伴——告诉他们你的专长和能力范围。你会得到免费的产品回报,有时候是生活中的。</p>
</li>
<li>
<p>容量是一个业务/产品问题。也就是说每个页面、上传或者登录等请求的网络消耗,都必须是可见的,以协助完成正确的业务/产品决策。</p>
</li>
<li>
<p>一定要打败预算!运维团队总是预算金额最大的挥霍者。公司的收入目标经常达不到,运维团队应该有很多办法来推迟自己的花费。</p>
</li>
<li>
<p>过去的经验不一定适用于现在乃至将来——多尝试没错,而且要有恰当的测试工具来做这件事。</p>
</li>
<li>
<p>文档——所有事情都应该好好记录成文档。避免团队的新成员绕着圈的找遍全团队逐一了解工作内容。</p>
</li>
<li>
<p>画一张超大尺寸的网络拓扑图,描绘你的数据中心。</p>
</li>
<li>
<p>为你的每个产品都画一个逻辑流程图。</p>
</li>
<li>
<p>维基——让大家可以很容易的发布“如何修复这个问题”的文档并且容易查找。这是技术作者发挥作用的地方,不过维基可以让哪怕非正式的文档或者增增改改的小段落也更好查看。</p>
</li>
<li>
<p>确保团队的每个成员,对,是每一个,都是可以替换的。</p>
</li>
<li>
<p>有些人在家里干活比在公司的时候还好,但有些人却不行。</p>
</li>
<li>
<p>订单打包签订——把硬件需求打包成大订单后再去咨询最大的折扣合同,记得订单里要包括所有一切,比如备件包,租赁条件等等。</p>
</li>
<li>
<p>和供应商保持长期联系,哪怕你换到下一份工作的时候也能联系上他们。</p>
</li>
<li>
<p>给运维团队每个人都配上一切他们可以远程操控的东西——掌上电脑, 3G 网卡,24 寸 LCD 屏幕……你为有才华的人付出得到的回报,远超过在远程雇佣的现场工程师。记住,运维工程师都是电力狂人,他们知道并且能充分利用屏幕上每个像素。</p>
</li>
<li>
<p>除非 Mac 可以运行 office 2007 和 outlook,否则团队里总需要几个 windows。这事很破坏团队的会议安排,联系人管理和邮件列表等等。</p>
</li>
<li>
<p>要有一个简化的采购流程——前提是你要了解自己的预算,并且能够管理好。我们可以从财务报告中得到实际。技术驱动的报告和财务驱动的报告之间通常存在差距。一个好的运维经理可以创建一些模型,将这些差别计入销售总成本中。而理解这些的 CFO 才可以帮助推动业务决策。</p>
</li>
<li>
<p>周会一定要持续举行,对上周的事件逐一总结和问责。</p>
</li>
<li>
<p>创建一个独立的升级系统,来管理那些对运维产生负面影响的代码开发工程。这个想法的来源是:一个同时涉及运维和开发的问题,在运维或者开发的跟踪系统里大多被湮没无视,最后没人理睬,所以给这些问题单独创建一个跟踪系统反而更加简单清楚。</p>
</li>
<li>
<p>产品开发从设计开始的每个阶段都要和运维技术相结合。这样,扩展性,监控和可靠性都融入到产品里。这样同时也可以确保运维负责的硬件采购、监控系统按时到位,运行手册即时更新,最后产品按照预计时间上线运行并且都符合运维标准。</p>
</li>
<li>
<p>像一个真正的公司一样运作——萨班斯法案,WebTrust 安全审计认证,SAS 70 审计标准,Visa 组织和银行等等。如果你真的成功了,这些都是你不得不打交道的。早点开始这些准备其实很简单,不需要太多的知识。不过就是开发一个工单/任务跟踪工具,然后好好使用。把变更控制和管理放进同样的系统里,好好使用。其他信息也放进来。系统就可以帮助我们找出像“上周变更了什么”这类信息。</p>
</li>
<li>
<p>给冗余留空间。一开始或许很难,但是一个没有真正的扩展性和可靠性的系统,才会真正耽误你获得成功的时间。</p>
</li>
<li>
<p>买个 Oracle 标准版(或者微软 SQL Server 标准版)是值得的。如果你可以限制住自己不超过标准版的需求,那就绝对值得买,哪怕你刚刚开始创业。</p>
</li>
<li>
<p>Postgres 和 MySQL 的免费不错。如果你不是特别在意事务完整性,MySQL 其实挺好的。</p>
</li>
<li>
<p>容量设计应该按照每日峰值再上抛 20% 到 30% 的冗余。除非你是个 vmotion(译注:VMWare 的热迁移技术)达人。</p>
</li>
<li>
<p>尽量多读一些贸易杂志。它们通常是免费的,只要你填写一些调查问卷就好了。新闻的价值是巨大的。对了,记得让他们投递到你家里,工作的时候读杂志的机会趋近于零。</p>
</li>
<li>
<p>注意安全。开发人员不应该有生产线的权限,而应该去做代码复核。这是和运维之间的职责分离。然后运维中应该有人控制设置其他运维人员权限的权限。创建一个员工手册,警告大家违反安全条例会有很严重的后果。从一开始就要记住从物理的、逻辑的、功能的各个方面来保护客户的数据安全和隐私。万一有客户要和你对簿公堂,你回忆起来发现自己只是靠勇气和勤奋来保护客户数据,这感觉可不怎么好。</p>
</li>
<li>
<p>控制好访问入口。首先要保证大家可以正常完成工作;其次要确保你知道他们是从哪里进来的。快去实现双因素身份验证方法吧。</p>
</li>
<li>
<p>对于人们访问生产环境必经之路的堡垒机和网关,键盘记录是至关重要的。对于 Windows 可能稍微有点难度,不过有些网关可以提供自动截屏功能。</p>
</li>
<li>
<p>确保有多种办法登录生产环境。不要期望公司的 VPN 在网络中断的时候还能起作用。直接把 VPN 架设在生产环境里。</p>
</li>
<li>
<p>使用 LDAP 做认证,哪怕你只有 10 台机器,通过复制 passwd 和 shadow 文件的方式来管理,你也要 LDAP 认证。</p>
</li>
<li>
<p>不要低估在 UNIX 环境中一台 Windows Server 2008 设备是多么有用。如果只是因为不懂 Windows,那么去学,而不是贬低它。</p>
</li>
<li>
<p>不要用那些无效的无线方案浪费大家的时间。公司里所有人都在移动,沙发上,会议室里,门口,到处都要上网。千万维护好你的无线路由。</p>
</li>
<li>
<p>总有些人把额外的精力和时间都投入到工作上——直接通过他们的请假单好了。而另一些人恰恰相反只把注意力放在怎么通过自己的请假单。在个人时间安排上,运维人员总是做出巨大的牺牲,他们随时准备凌晨3点爬起床快速响应排障需求。</p>
</li>
<li>
<p>通过集中式的 RDBMS 管理你所有的设备资产。然后复制资产,人员,网络,合同等所有数据到异地。没错,要的是一个在线的实时可用的复制,而不是每天晚上备份到磁带。</p>
</li>
<li>
<p>自动使用多进程以确认安全,包括操作系统或者产品的上线,文件的推送,日志的分析等。</p>
</li>
<li>
<p>自动化操作必须和运维的 RDBMS 数据相关联。</p>
</li>
<li>
<p>设备通常有三种状态——离线,服务中,预备。预备状态就是说正在通过 cfengine、rsync 或者其他你在使用的工具完成配置。服务中就是已经运行着流量了。同时还需要一个状态,这个状态下的设备可以在不提供生产服务的情况下收集或者测试数据。</p>
</li>
<li>
<p>尊重日志数据。在设备下线或者重建之前,一定要先导出日志。</p>
</li>
<li>
<p>如果业务飞速发展让你没有太多时间来做优化,那就尽力锁定一切——进程还能工作,就不要改变它,直到后来有了绝对必要的理由。总之,锁定默认值,等待成长到必要时再审视。</p>
</li>
<li>
<p>你永远无法避免运维工程师在你基础设施最关键的地方犯点啥错——比如在哪台机器上不小心执行 <code class="highlighter-rouge">rm -rf /</code> 命令。</p>
</li>
<li>
<p>为团队保持好玩和有趣的气氛——如果他们不再享受他们的工作,他们就会找别的事情来消遣。要让团队有主人翁意识,运维不是哪个经理的个人任务。</p>
</li>
<li>
<p>提供 99.999% 可用性的真正价值在于让我们有能力保持灵活。这意味着当你需要的时候可以充分利用系统冗余。物理变更、设备迁移、代码修改和回退等等都游刃有余。这个对于公司本身价值巨大,甚至比对客户还大。</p>
</li>
<li>
<p>如果你能做到 99.999%,那就给客户一个 100% 的SLA承诺。</p>
</li>
<li>
<p>不要湮没软件热更新的能力。应该被湮没的是你自己回滚或者转移到旧版本代码的能力。压根就不应该“处理”这种徒劳的失败转移。当事情变得不如人意的时候,你更应该做的是找个大玩意儿来挡住你的肥屁股。CYA(译注:Cover Your Ass,就是前面说的盖屁股) = 保持敏捷 = 成功的公司。</p>
</li>
<li>
<p>记住你为客户构建产品的思路里每一步的原因和目的——不管你部署给最终用户的是什么,把这些放在最先考虑,即你所有(基础设施、流程和人员)的设计都是为了提供最好的服务和产品。</p>
</li>
<li>
<p>第一次就要成功。很少有机会让你回去重新开始的。重做是对公司资源的巨大浪费。</p>
</li>
<li>
<p>多联系业内的合作伙伴、盟友和类似的企业,看看他们的运维是怎么做的。很可能他们碰到了跟你一样的挑战,而解决的更为巧妙。不要害怕分享自己的经验和处理过程,因为别人也会回馈的。</p>
</li>
<li>
<p>招人就要招那些足以让自己担心会被挤掉目前工作的,招那些你欣赏和可以学习的榜样,招那些你愿意和他一起工作的。这感觉甚至超过你招聘一个工作考评为A的员工。</p>
</li>
<li>
<p>IT 和运维是完全不同的两个概念。一个不错的运维经理应该可以管理好企业 IT,但是一个传统的 IT 工程师很难有能力处理互联网运维任务。</p>
</li>
<li>
<p>当你开始一份新工作或者在每年的起始,都应该去争取预算。这不是说滚着那个滋滋响的轮子往前走(应该是指循规蹈矩照本宣科),而是要一个基于历史数据做出的优秀的文案。如果你正在评估一份新工作,请确认你完完全全的知道预算以及预算的来源。同时,还应该有的是改善这份预算的权利。</p>
</li>
</ol>
puppet和rex的常用资源写法类比
2013-05-28T00:00:00+08:00
devops
rex
puppet
http://chenlinux.com/2013/05/28/compare-dsl-of-puppet-with-rex
<p>首先要申明,rex 和 puppet 本质上是不同的,puppet 追求的是状态,rex 追求的是操作。puppet 用户经常关心的是 agent 运行了没,而 rex 用户关心的是怎么写 Rexfile 能让中控运行 rex 时的命令参数更简洁漂亮(个人感受==!)。所以哪怕在本文中列举的这些资源写法很类似,也请读者们注意:rex 的资源关键词命名,都是带有动作性的,比如 <code class="highlighter-rouge">create</code>,<code class="highlighter-rouge">add</code>,<code class="highlighter-rouge">install</code>,<code class="highlighter-rouge">upload</code>,<code class="highlighter-rouge">download</code>,<code class="highlighter-rouge">sync</code> 等等。甚至精确的说,rex 里这些不是资源(<code class="highlighter-rouge">Puppet::Types::***</code>),他们是 <code class="highlighter-rouge">Rex::Commands::***</code>。</p>
<p>因为 rex 基于并发 ssh 连接,所以它有一些操作是 puppet 所没有的,比如 <code class="highlighter-rouge">tail</code>,<code class="highlighter-rouge">file_append</code>,<code class="highlighter-rouge">fdisk</code>,<code class="highlighter-rouge">sysctl</code> 和 <code class="highlighter-rouge">iptables</code> 等等,这里暂时不列举。总的来说,本文目的是总结类似的部分,而不是不同的用法……</p>
<h1 id="cron-">Cron 资源</h1>
<h3 id="puppet-">puppet 写法</h3>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">cron</span> <span class="p">{</span> <span class="s1">'check_starttime'</span><span class="p">:</span>
<span class="k">ensure</span> <span class="o">=></span> <span class="n">present</span><span class="p">,</span>
<span class="n">minute</span> <span class="o">=></span> <span class="mi">30</span><span class="p">,</span>
<span class="n">hour</span> <span class="o">=></span> <span class="s1">'*/2'</span><span class="p">,</span>
<span class="n">user</span> <span class="o">=></span> <span class="s1">'root'</span><span class="p">,</span>
<span class="n">command</span> <span class="o">=></span> <span class="s1">'sh /usr/local/bin/check_start_time.sh'</span><span class="p">,</span>
<span class="nb">require</span> <span class="o">=></span> <span class="no">File</span><span class="p">[</span><span class="s1">'/usr/local/bin/check_start_time.sh'</span><span class="p">],</span>
<span class="p">}</span>
</code></pre>
</div>
<h3 id="rex-">rex 写法</h3>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">cron</span> <span class="nv">add</span> <span class="o">=></span> <span class="s">"root"</span><span class="p">,</span> <span class="p">{</span>
<span class="nv">minute</span> <span class="o">=></span> <span class="s">'5'</span><span class="p">,</span>
<span class="nv">hour</span> <span class="o">=></span> <span class="s">'*'</span><span class="p">,</span>
<span class="nv">day_of_month</span> <span class="o">=></span> <span class="s">'*'</span><span class="p">,</span>
<span class="nv">month</span> <span class="o">=></span> <span class="s">'*'</span><span class="p">,</span>
<span class="nv">day_of_week</span> <span class="o">=></span> <span class="s">'*'</span><span class="p">,</span>
<span class="nv">command</span> <span class="o">=></span> <span class="s">'/path/to/your/cronjob'</span><span class="p">,</span>
<span class="p">};</span>
</code></pre>
</div>
<h1 id="file-">File 资源</h1>
<h3 id="puppet--1">puppet 写法</h3>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">file</span> <span class="p">{</span> <span class="s1">'/etc/squid/squid.conf'</span><span class="p">:</span>
<span class="k">ensure</span> <span class="o">=></span> <span class="n">file</span><span class="p">,</span>
<span class="n">mode</span> <span class="o">=></span> <span class="s1">'0755'</span><span class="p">,</span>
<span class="n">content</span> <span class="o">=></span> <span class="n">template</span><span class="p">(</span><span class="s1">'squid/squid.conf.erb'</span><span class="p">),</span>
<span class="nb">require</span> <span class="o">=></span> <span class="no">Package</span><span class="p">[</span><span class="s1">'squid'</span><span class="p">],</span>
<span class="n">subscribe</span> <span class="o">=></span> <span class="no">Service</span><span class="p">[</span><span class="s1">'squid'</span><span class="p">],</span>
<span class="p">}</span>
</code></pre>
</div>
<h3 id="rex--1">rex 写法</h3>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">file</span> <span class="s">"/etc/squid/squid.conf"</span><span class="p">,</span>
<span class="nv">content</span> <span class="o">=></span> <span class="nv">template</span><span class="p">(</span><span class="s">"templates/squid.tpl"</span><span class="p">,</span> <span class="nv">vars</span> <span class="o">=></span> <span class="o">\</span><span class="nv">%var</span> <span class="p">),</span>
<span class="nv">owner</span> <span class="o">=></span> <span class="s">"root"</span><span class="p">,</span>
<span class="nv">group</span> <span class="o">=></span> <span class="s">"root"</span><span class="p">,</span>
<span class="nv">mode</span> <span class="o">=></span> <span class="mi">700</span><span class="p">,</span>
<span class="nv">needs</span> <span class="o">=></span> <span class="nv">SquidPkgTask</span><span class="p">,</span>
<span class="nv">on_change</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="nv">service</span> <span class="nv">squid</span> <span class="o">=></span> <span class="s">'restart'</span> <span class="p">};</span>
</code></pre>
</div>
<p>这里的 <code class="highlighter-rouge">on_change</code> 是 File 资源独有的。</p>
<p><strong>通用资源方面,rex 中在同一个 task 内,是按照书写顺序执行;在 task 之间,通过 <code class="highlighter-rouge">needs</code> 可以定义依赖。</strong></p>
<p>另外 rex 还有 <code class="highlighter-rouge">before</code>,<code class="highlighter-rouge">after</code>,<code class="highlighter-rouge">around</code> 三个关键字作用于 task 上。不过这三个是在 rex 控制端执行,不是在远端主机上执行。</p>
<p>注意这里,这个 file 看起来没有使用操作性的动词,但其实他是下面这个写法的简写而已:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">install</span> <span class="nv">file</span> <span class="o">=></span> <span class="s">'templates/etc/hosts.tpl'</span><span class="p">,</span> <span class="p">{</span>
<span class="nv">source</span> <span class="o">=></span> <span class="s">"/etc/hosts"</span><span class="p">,</span>
<span class="nv">owner</span> <span class="o">=></span> <span class="s">"root"</span><span class="p">,</span>
<span class="nv">group</span> <span class="o">=></span> <span class="s">"root"</span><span class="p">,</span>
<span class="nv">mode</span> <span class="o">=></span> <span class="mi">700</span><span class="p">,</span>
<span class="nv">on_change</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="nv">say</span> <span class="s">"Something was changed."</span> <span class="p">},</span>
<span class="nv">template</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">greeting</span> <span class="o">=></span> <span class="s">"hello"</span><span class="p">,</span>
<span class="nv">name</span> <span class="o">=></span> <span class="s">"Ben"</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">};</span>
</code></pre>
</div>
<p>另外,还有一个通过 SFTP 接口上传的写法:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">upload</span> <span class="s">"hosts"</span> <span class="o">=></span> <span class="s">"/etc/"</span><span class="p">;</span>
</code></pre>
</div>
<h1 id="package-">Package 资源</h1>
<h3 id="puppet--2">puppet 写法</h3>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">package</span> <span class="p">{</span> <span class="s1">'ganglia-gmond-modules-python-plugin'</span><span class="p">:</span>
<span class="k">ensure</span> <span class="o">=></span> <span class="n">installed</span><span class="p">,</span>
<span class="nb">require</span> <span class="o">=></span> <span class="no">Class</span><span class="p">[</span><span class="s1">'repos'</span><span class="p">],</span>
<span class="p">}</span>
</code></pre>
</div>
<h3 id="rex--2">rex 写法</h3>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">repository</span> <span class="nv">add</span> <span class="o">=></span> <span class="nv">myrepo</span><span class="p">,</span>
<span class="nv">url</span> <span class="o">=></span> <span class="s">'http://rex.linux-files.org/CentOS/$releasever/rex/$basearch/'</span><span class="p">;</span>
<span class="nv">update_package_db</span><span class="p">;</span>
<span class="nv">install</span> <span class="nb">package</span> <span class="o">=></span> <span class="s">'vim'</span><span class="p">;</span>
</code></pre>
</div>
<h1 id="class-">Class 定义</h1>
<h3 id="puppet--3">puppet 写法</h3>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">class</span> <span class="n">squid</span> <span class="p">{</span>
<span class="kp">include</span> <span class="n">squid</span><span class="o">::</span><span class="n">install</span>
<span class="p">}</span>
</code></pre>
</div>
<h3 id="rex--3">rex 写法</h3>
<p>rex 执行的 Rexfile 其实就是 perl 的模块文件,所以写法就是标准的 perl 写法。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">package</span> <span class="nv">Squid</span> <span class="p">{</span>
<span class="nb">require</span> <span class="nn">Squid::</span><span class="nv">Install</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
<p>呼呼,新版本的 Perl 中可以用 <code class="highlighter-rouge"><span class="p">{}</span></code> 来包裹 package 定义的内容,看起来是不是更像一些?不过 CentOS6 的 5.10 版还不支持,所以通用起见,还是这样写吧:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">package</span> <span class="nv">Squid</span><span class="p">;</span>
<span class="nb">require</span> <span class="nn">Squid::</span><span class="nv">Install</span><span class="p">;</span>
<span class="mi">1</span><span class="p">;</span>
</code></pre>
</div>
<h1 id="directory-">Directory 资源</h1>
<h3 id="puppet--4">puppet 写法</h3>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">file</span> <span class="p">{</span> <span class="s1">'murder-client'</span><span class="p">:</span>
<span class="k">ensure</span> <span class="o">=></span> <span class="s1">'directory'</span><span class="p">,</span>
<span class="n">path</span> <span class="o">=></span> <span class="s1">'/usr/local/murder'</span><span class="p">,</span>
<span class="n">recurse</span> <span class="o">=></span> <span class="kp">true</span><span class="p">,</span>
<span class="n">purge</span> <span class="o">=></span> <span class="kp">true</span><span class="p">,</span>
<span class="n">source</span> <span class="o">=></span> <span class="s1">'puppet:///modules/murder/dist'</span><span class="p">,</span>
<span class="p">}</span>
</code></pre>
</div>
<h3 id="rex--4">rex 写法</h3>
<p>rex 中采用 rsync 来完成目录文件的同步:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">mkdir</span><span class="p">(</span><span class="s">'/usr/local/murder'</span><span class="p">);</span>
<span class="nv">sync</span> <span class="s">'dist/*'</span> <span class="o">=></span> <span class="s">'/usr/local/murder'</span><span class="p">,</span> <span class="p">{</span>
<span class="nv">exclude</span> <span class="o">=></span> <span class="s">"*.sw*"</span><span class="p">,</span>
<span class="nv">parameters</span> <span class="o">=></span> <span class="s">'--backup --delete'</span><span class="p">,</span>
<span class="p">};</span>
</code></pre>
</div>
<h1 id="shell-">Shell 资源</h1>
<h3 id="puppet--5">puppet 写法</h3>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">exec</span> <span class="p">{</span><span class="s1">'init-reload'</span><span class="p">:</span>
<span class="n">command</span> <span class="o">=></span> <span class="s1">'/sbin/initctl reload-configuration && /sbin/initctl start svscan'</span><span class="p">,</span>
<span class="n">subscribe</span> <span class="o">=></span> <span class="no">File</span><span class="p">[</span><span class="s1">'/etc/init/svscan.conf'</span><span class="p">],</span>
<span class="n">refreshonly</span> <span class="o">=></span> <span class="kp">true</span><span class="p">,</span>
<span class="p">}</span>
</code></pre>
</div>
<h3 id="rex--5">rex 写法</h3>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">run</span> <span class="s">"cmd"</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$out</span><span class="p">,</span> <span class="nv">$err</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<p>这个回调函数可以不要,那么 <code class="highlighter-rouge">run</code> 命令返回输出到变量。这种用法在单行命令中最常用,比如这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> rex -H <span class="s1">'192.168.0.[10..30]'</span> -e <span class="s1">'say run "df -h"'</span>
</code></pre>
</div>
<h1 id="usergroup-">User/Group 资源</h1>
<h3 id="puppet--6">puppet 写法</h3>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">group</span> <span class="p">{</span><span class="s1">'puppet'</span><span class="p">:</span>
<span class="k">ensure</span> <span class="o">=></span> <span class="n">present</span><span class="p">,</span>
<span class="n">gid</span> <span class="o">=></span> <span class="mi">501</span><span class="p">,</span>
<span class="nb">system</span> <span class="o">=></span> <span class="kp">true</span><span class="p">,</span>
<span class="p">}</span>
<span class="n">user</span> <span class="p">{</span><span class="s1">'puppet'</span><span class="p">:</span>
<span class="k">ensure</span> <span class="o">=></span> <span class="n">present</span><span class="p">,</span>
<span class="n">uid</span> <span class="o">=></span> <span class="mi">501</span><span class="p">,</span>
<span class="nb">system</span> <span class="o">=></span> <span class="kp">true</span><span class="p">,</span>
<span class="n">groups</span> <span class="o">=></span> <span class="p">[</span><span class="s1">'puppet'</span><span class="p">,</span> <span class="s1">'...'</span><span class="p">],</span>
<span class="n">expiry</span> <span class="o">=></span> <span class="s1">'2013-05-30'</span><span class="p">,</span>
<span class="n">managehome</span> <span class="o">=></span> <span class="kp">false</span><span class="p">,</span>
<span class="p">}</span>
</code></pre>
</div>
<h3 id="rex--6">rex 写法</h3>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">create_group</span> <span class="s">'puppet'</span><span class="p">,</span> <span class="p">{</span>
<span class="nv">gid</span> <span class="o">=></span> <span class="mi">501</span><span class="p">,</span>
<span class="nb">system</span> <span class="o">=></span> <span class="mi">1</span><span class="p">,</span>
<span class="p">}</span>
<span class="nv">create_user</span> <span class="s">'puppet'</span><span class="p">,</span>
<span class="nv">uid</span> <span class="o">=></span> <span class="mi">501</span><span class="p">,</span>
<span class="nv">home</span> <span class="o">=></span> <span class="s">'/etc/puppet'</span><span class="p">,</span>
<span class="nv">expire</span> <span class="o">=></span> <span class="s">'2013-05-30'</span><span class="p">,</span>
<span class="nv">groups</span> <span class="o">=></span> <span class="p">[</span><span class="s">'puppet'</span><span class="p">,</span> <span class="s">'...'</span><span class="p">],</span>
<span class="nv">password</span> <span class="o">=></span> <span class="s">'blahblah'</span><span class="p">,</span>
<span class="nb">system</span> <span class="o">=></span> <span class="mi">1</span><span class="p">,</span>
<span class="nv">no_create_home</span> <span class="o">=></span> <span class="nv">TRUE</span><span class="p">,</span>
<span class="nv">ssh_key</span> <span class="o">=></span> <span class="s">"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQChUw..."</span><span class="p">;</span>
</code></pre>
</div>
<h1 id="service-">Service 资源</h1>
<h3 id="puppet--7">puppet 写法</h3>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">service</span> <span class="p">{</span><span class="s1">'nginx'</span><span class="p">:</span>
<span class="k">ensure</span> <span class="o">=></span> <span class="kp">true</span><span class="p">,</span>
<span class="n">enable</span> <span class="o">=></span> <span class="kp">true</span><span class="p">,</span>
<span class="p">}</span>
</code></pre>
</div>
<h3 id="rex--7">rex 写法</h3>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">service</span> <span class="nv">apache2</span> <span class="o">=></span> <span class="nv">ensure</span> <span class="o">=></span> <span class="s">"started"</span><span class="p">;</span>
<span class="nv">service</span> <span class="nv">apache2</span> <span class="o">=></span> <span class="s">"start"</span><span class="p">;</span>
</code></pre>
</div>
<p>再次可见,rex 认为 <code class="highlighter-rouge">service</code> 命令和 <code class="highlighter-rouge">chkconfig</code>/<code class="highlighter-rouge">update-rc.d</code> 命令是两件事情,所以要分开两个写法。</p>
<h1 id="mount-">Mount 资源</h1>
<h3 id="puppet--8">puppet 写法</h3>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">mount</span> <span class="p">{</span><span class="s1">'/mnt/sda6'</span><span class="p">:</span>
<span class="k">ensure</span> <span class="o">=></span> <span class="n">present</span><span class="p">;</span>
<span class="n">device</span> <span class="o">=></span> <span class="s1">'/dev/sda6'</span><span class="p">,</span>
<span class="n">fstype</span> <span class="o">=></span> <span class="s1">'ext3'</span><span class="p">,</span>
<span class="n">options</span> <span class="o">=></span> <span class="s1">'noatime,async'</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
<h3 id="rex--8">rex 写法</h3>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">mount</span> <span class="s">"/dev/sda6"</span><span class="p">,</span> <span class="s">"/mnt/sda6"</span><span class="p">,</span>
<span class="nv">fs</span> <span class="o">=></span> <span class="s">"ext3"</span><span class="p">,</span>
<span class="nv">options</span> <span class="o">=></span> <span class="p">[</span><span class="sx">qw/noatime async/</span><span class="p">];</span>
</code></pre>
</div>
<h1 id="facts-">Facts 变量和模板</h1>
<h3 id="puppet--9">puppet 写法</h3>
<p>在 puppet 中,Facts 变量有两种用法,一个是 <code class="highlighter-rouge">*.pp</code> 里的写法:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="vg">$:</span><span class="ss">:lsbdistid</span>
</code></pre>
</div>
<p>另一种是在 <code class="highlighter-rouge">*.erb</code> 里的写法,值得注意的是变量的作用域:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="o"><</span><span class="sx">%= scope::lookupvar('ipaddress') %>
<%=</span> <span class="n">scope</span><span class="o">::</span><span class="n">lookupvar</span><span class="p">(</span><span class="s1">'nginx::name'</span><span class="p">)</span> <span class="o">%></span>
</code></pre>
</div>
<h3 id="rex--9">rex 写法</h3>
<p>在 rex 中,远端主机的系统状态有多种获取方式,比如:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="c1"># 全部,这些变量默认会传递给 template</span>
<span class="k">my</span> <span class="nv">$sysinfo</span> <span class="o">=</span> <span class="nn">Rex::Helper::System::</span><span class="nv">info</span><span class="p">;</span>
<span class="c1"># 实际就是从上面info里取具体的变量</span>
<span class="k">my</span> <span class="nv">$lsd</span> <span class="o">=</span> <span class="nv">get_operating_system</span><span class="p">;</span>
<span class="c1"># 这个慎用,会死人的</span>
<span class="k">my</span> <span class="nv">@ns</span> <span class="o">=</span> <span class="nv">netstat</span><span class="p">;</span>
</code></pre>
</div>
<p>也可以使用 <code class="highlighter-rouge">set</code> 指令,这种变量和使用 perl 标准 <code class="highlighter-rouge">my $name</code> 方式不同的是它可以直接在模板中读取:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">set</span> <span class="nv">name</span> <span class="o">=></span> <span class="s">'CDN'</span><span class="p">;</span>
</code></pre>
</div>
<p>至于 rex 的模板,它默认没有使用 CPAN 上任何一种现成的模块,而是自己实现了一个,写法如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">template</span><span class="p">(</span><span class="s">'your.tpl'</span><span class="p">,</span> <span class="nv">yourvars</span> <span class="o">=></span> <span class="o">\</span><span class="nv">%hash</span> <span class="p">);</span>
</code></pre>
</div>
<p>然后在模板中这样引用:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">My</span> <span class="nv">variable</span> <span class="nv">is</span> <span class="o"><</span><span class="nv">%</span><span class="err">=</span> <span class="err">$</span><span class="nv">::yourvars</span><span class="o">-></span><span class="p">{</span><span class="nv">key</span><span class="p">}</span> <span class="nv">%</span><span class="err">></span>
<span class="nv">My</span> <span class="nv">name</span> <span class="nv">is</span> <span class="o"><</span><span class="nv">%</span><span class="err">=</span> <span class="err">$</span><span class="nv">::name</span> <span class="nv">%</span><span class="err">></span>
<span class="nv">My</span> <span class="nv">lsd</span> <span class="nv">is</span> <span class="o"><</span><span class="nv">%</span><span class="err">=</span> <span class="err">$</span><span class="nv">::operatingsystem</span> <span class="nv">%</span><span class="err">></span>
</code></pre>
</div>
<p>明显有模仿 puppet 的痕迹,传递进模版的变量以 <code class="highlighter-rouge">$::</code> 开头,个人比较汗……</p>
<p>所以个人建议还是更换成 CPAN 上的流行模板,比如 <code class="highlighter-rouge">Text::Xslate</code> 或者 <code class="highlighter-rouge">Text::MicroTemplate</code> 等等,使用 <code class="highlighter-rouge">set_template_option</code> 即可。</p>
使用 Rex::Box 代替 Vagrant 的工作
2013-05-27T00:00:00+08:00
devops
rex
perl
virtualbox
http://chenlinux.com/2013/05/27/use-rex-box-replace-vagrant
<p>Vagrant 是近来 devops 界内非常流行和火爆的工具,它和 puppet/chef 的结合,成为运维开发和测试,甚至预热部署的重要手段。比如在 cloudfoundry 官方放弃使用 <code class="highlighter-rouge">vcap_setup</code> 脚本部署后,社区大多对其 <code class="highlighter-rouge">BOSH</code> 不买账,转而研究使用 vagrant 部署了。</p>
<p>对于 perl 运维人员,使用 Rex 工具做集群管理的话,其实完全不用再使用 vagrant 了。因为 Rex 自带有 Box 功能。完全可以一体化工作。下面从 Rex 官网上半翻译半截取两篇文章,展示 Rex::Box 的使用。两篇原文分别是:</p>
<ol>
<li><a href="http://box.rexify.org/guide">http://box.rexify.org/guide</a></li>
<li><a href="http://www.rexify.org/howtos/use_boxes_with_any_box_provider.html">http://www.rexify.org/howtos/use_boxes_with_any_box_provider.html</a></li>
</ol>
<h1 id="section">环境准备</h1>
<div class="highlighter-rouge"><pre class="highlight"><code>rexify <span class="nv">$project</span>-name --template box
<span class="nb">cd</span> <span class="nv">$project</span>-name
rex init --name<span class="o">=</span><span class="nv">$vm</span>-name --url<span class="o">=</span><span class="nv">$url</span>-to-prebuild-vm-image
</code></pre>
</div>
<h1 id="section-1">虚拟机定义</h1>
<p>这里有两种方式,一种是类似 Vagrantfile 定义的 Rexfile 写法:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nv">set</span> <span class="nv">box</span> <span class="o">=></span> <span class="s">"VBox"</span><span class="p">;</span>
<span class="nv">task</span> <span class="nv">mytask</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">box</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$box</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="nv">$box</span><span class="o">-></span><span class="nv">name</span><span class="p">(</span><span class="s">"boxname"</span><span class="p">);</span>
<span class="nv">$box</span><span class="o">-></span><span class="nv">url</span><span class="p">(</span><span class="s">"http://box.rexify.org/box/base-image.box"</span><span class="p">);</span>
<span class="nv">$box</span><span class="o">-></span><span class="nv">network</span><span class="p">(</span><span class="mi">1</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">type</span> <span class="o">=></span> <span class="s">"bridged"</span> <span class="c1"># 默认是 "nat",</span>
<span class="nv">bridge</span> <span class="o">=></span> <span class="s">"eth0"</span><span class="p">,</span>
<span class="p">});</span>
<span class="nv">$box</span><span class="o">-></span><span class="nv">forward_port</span><span class="p">(</span><span class="nv">ssh</span> <span class="o">=></span> <span class="p">[</span><span class="mi">2222</span><span class="p">,</span> <span class="mi">22</span><span class="p">]);</span>
<span class="nv">$box</span><span class="o">-></span><span class="nv">share_folder</span><span class="p">(</span><span class="nv">boxhome</span> <span class="o">=></span> <span class="s">"/path/to/myuser"</span><span class="p">);</span>
<span class="nv">$box</span><span class="o">-></span><span class="nv">auth</span><span class="p">(</span>
<span class="nv">user</span> <span class="o">=></span> <span class="s">"root"</span><span class="p">,</span>
<span class="nv">password</span> <span class="o">=></span> <span class="s">"box"</span><span class="p">,</span>
<span class="p">);</span>
<span class="nv">$box</span><span class="o">-></span><span class="nv">setup</span><span class="p">(</span><span class="sx">qw/setup_frontend/</span><span class="p">);</span>
<span class="p">};</span>
<span class="p">};</span>
</code></pre>
</div>
<p>另一种是采用 YAML 配置:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="s">type</span><span class="pi">:</span> <span class="s">VBox</span>
<span class="s">vms</span><span class="pi">:</span>
<span class="s">fe01</span><span class="pi">:</span>
<span class="s">url</span><span class="pi">:</span> <span class="s">http://box.rexify.org/box/ubuntu-server-12.10-amd64.ova</span>
<span class="s">network</span><span class="pi">:</span>
<span class="s">1</span><span class="pi">:</span>
<span class="s">type</span><span class="pi">:</span> <span class="s">bridged</span>
<span class="s">bridge</span><span class="pi">:</span> <span class="s">eth0</span>
<span class="s">setup</span><span class="pi">:</span> <span class="s">setup_frontend</span>
<span class="s">db01</span><span class="pi">:</span>
<span class="s">url</span><span class="pi">:</span> <span class="s">http://box.rexify.org/box/ubuntu-server-12.10-amd64.ova</span>
<span class="s">network</span><span class="pi">:</span>
<span class="s">1</span><span class="pi">:</span>
<span class="s">type</span><span class="pi">:</span> <span class="s">bridged</span>
<span class="s">bridge</span><span class="pi">:</span> <span class="s">eth0</span>
<span class="s">setup</span><span class="pi">:</span> <span class="s">setup_db</span>
</code></pre>
</div>
<h1 id="section-2">虚拟机初始化</h1>
<p>在 Vagrant 中有一个概念叫 provision,也就是在虚拟机第一次运行时,通过 shell/puppet/chef 等进行初始化操作。Rex::Box 自然是通过 Rex 本身来进行这个任务。也就是上例中的 <code class="highlighter-rouge">setup</code> 定义的 task 名称。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nv">task</span> <span class="s">'setup_frontend'</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">install</span> <span class="nv">nginx</span><span class="p">;</span>
<span class="nv">file</span> <span class="s">'/etc/nginx.conf'</span><span class="p">,</span>
<span class="nv">content</span> <span class="o">=></span> <span class="nv">template</span><span class="p">(</span><span class="s">'template/httpd.conf.tpl'</span><span class="p">),</span>
<span class="nv">owner</span> <span class="o">=></span> <span class="s">"root"</span><span class="p">,</span>
<span class="nv">group</span> <span class="o">=></span> <span class="s">"root"</span><span class="p">,</span>
<span class="nv">on_change</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="nv">service</span> <span class="nv">nginx</span> <span class="o">=></span> <span class="s">"restart"</span><span class="p">;</span> <span class="p">};</span>
<span class="p">};</span>
</code></pre>
</div>
<p>因为 rex 本身是通过 ssh 管理,所以在 setup 之前,必须定义好如何 auth,自己做的镜像不说了,通过 rexify.org 下载的默认镜像,就是默认的 root/box 了。</p>
<p>说到镜像,其实 vagrant 的 <code class="highlighter-rouge">.box</code> 也就是 <code class="highlighter-rouge">.ova</code> ,都是把 virtualbox 的 <code class="highlighter-rouge">.vmdk</code> 和 <code class="highlighter-rouge">.ovf</code> 打了个包而已。</p>
<p>当然,也可以在 task 写 shell,通过 <code class="highlighter-rouge">run</code> 的方式,其实 run 应该也是 Rex 最常用的 task 了。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nv">task</span> <span class="s">'setup_frontend'</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">run</span> <span class="s">"echo Hello, world"</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<h1 id="section-3">虚拟机使用</h1>
<p>定义完成后,就可以使用 init 配置虚拟机环境,然后 start/stop 管理虚拟机。</p>
<p>比如在使用 YAML 配置的时候,配置环境的 Rexfile 最后是这样的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nn">Rex::Commands::</span><span class="nv">Box</span> <span class="nv">init_file</span> <span class="o">=></span> <span class="s">"box.yml"</span><span class="p">;</span>
<span class="nv">group</span> <span class="nv">myboxes</span> <span class="o">=></span> <span class="nb">map</span> <span class="p">{</span> <span class="nv">get_box</span><span class="p">(</span><span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="nv">name</span><span class="p">})</span><span class="o">-></span><span class="p">{</span><span class="nv">ip</span><span class="p">}</span> <span class="p">}</span> <span class="nv">list_boxes</span><span class="p">;</span>
<span class="nv">task</span> <span class="s">"box"</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">boxes</span> <span class="s">"init"</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<p>像要做成命令行管理也比较简单,比如启动和停止虚拟机的 task 这样写:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nv">task</span> <span class="s">"stop"</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$param</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="nv">boxes</span> <span class="nv">stop</span> <span class="o">=></span> <span class="nv">$param</span><span class="o">-></span><span class="p">{</span><span class="nv">name</span><span class="p">};</span>
<span class="p">};</span>
</code></pre>
</div>
<p>就可以在命令行直接这样启动某个虚拟机了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>rex stop --name<span class="o">=</span>myvbox
</code></pre>
</div>
<p>事实上,本文最开头的默认 box 模板生成的命令,就是通过前一步生成的 Rexfile 里定义的 <code class="highlighter-rouge">task "init", sub {...};</code> 实现的。</p>
<p><strong>2013 年 07 月 23 日附注:</strong></p>
<p>虽然如此,但是 Vagrant 目前已经成为开源社区风头正劲的一个产品,其开放的 plugin 机制导致周边产品大量出现,已经形成了一个不错的社区氛围。还是建议大家了解 Vagrant 。目前 vagrant-plugin 列表见:<a href="https://github.com/mitchellh/vagrant/wiki/Available-Vagrant-Plugins">https://github.com/mitchellh/vagrant/wiki/Available-Vagrant-Plugins</a>。</p>
用mojo抓取数据并gocr替换图片内容
2013-05-14T00:00:00+08:00
perl
http://chenlinux.com/2013/05/14/mojo-and-gocr-for-buildhr-telephone
<p>现在的网站越来越狡猾,连招聘网站的信息都懂的把公司的联系方式动态图片化了。还好为了观看方便,没加什么干扰。所以写个脚本来识别还是可以的。虽然到目前为止没发现比较好的 OCR 工具——我指的是可以直接apt-get安装的,有朋友知道哪个比较好的话,欢迎告诉我~</p>
<p>尝试了一下 tesseract-ocr 和 gocr ,还是 gocr 靠谱一点点。所以 <code class="highlighter-rouge">apt-get install gocr</code> 安装然后运行下面这个 Perl 脚本:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nv">ojo</span><span class="p">;</span>
<span class="k">use</span> <span class="mf">5.010</span><span class="p">;</span>
<span class="nv">g</span><span class="p">(</span><span class="s">"http://search.buildhr.com/job/581968.html"</span><span class="p">)</span><span class="o">-></span><span class="nv">dom</span><span class="o">-></span><span class="nv">charset</span><span class="p">(</span><span class="s">"UTF-8"</span><span class="p">)</span><span class="o">-></span><span class="nv">find</span><span class="p">(</span><span class="s">"div .postjob .padding"</span><span class="p">)</span><span class="o">-></span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">-></span><span class="nv">find</span><span class="p">(</span><span class="s">"p"</span><span class="p">)</span><span class="o">-></span><span class="nb">each</span><span class="p">(</span><span class="nv">sub</span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$line</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$img_element</span> <span class="o">=</span> <span class="nv">$line</span><span class="o">-></span><span class="nv">at</span><span class="p">(</span><span class="s">'img'</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nb">defined</span> <span class="nv">$img_element</span><span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$img_url</span> <span class="o">=</span> <span class="nv">$img_element</span><span class="o">-></span><span class="p">{</span><span class="nv">src</span><span class="p">};</span>
<span class="nv">g</span><span class="p">(</span><span class="nv">$img_url</span><span class="p">)</span><span class="o">-></span><span class="nv">content</span><span class="o">-></span><span class="nv">asset</span><span class="o">-></span><span class="nv">move_to</span><span class="p">(</span><span class="s">"test.jpg"</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$seem_str</span> <span class="o">=</span> <span class="sb">`gocr test.jpg`</span><span class="p">;</span>
<span class="nb">chomp</span><span class="p">(</span><span class="nv">$seem_str</span><span class="p">);</span>
<span class="nv">say</span> <span class="nb">join</span><span class="p">(</span><span class="nv">$seem_str</span><span class="p">,</span> <span class="nb">split</span><span class="p">(</span><span class="sr">/ /</span><span class="p">,</span> <span class="nv">$line</span><span class="o">-></span><span class="nv">text</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">});</span>
</code></pre>
</div>
<p>不过老是把 <code class="highlighter-rouge">7</code> 识别成 <code class="highlighter-rouge">_</code>。</p>
<p>真是越来越觉得 ojo 好用啊~</p>
Newbie::Gift 所用知识总结
2013-04-19T00:00:00+08:00
perl
http://chenlinux.com/2013/04/19/something-dive-to-perl
<p>通过 <a href="https://github.com/chenryn/Newbie-Gift">Newbie::Gift</a> 项目的开发过程,学习和深入了解了不少 Perl 知识,虽然这个模块估计短期内不会再继续开发和更新了,不过还是值得记录一下这段过程中的心得。</p>
<h3 id="gensym">gensym</h3>
<p>封装 <code class="highlighter-rouge">IPC::Open3</code> 模块时,通过 <code class="highlighter-rouge">smokeping</code> 代码中学到了 <code class="highlighter-rouge">Symbol</code> 模块的 <code class="highlighter-rouge">gensym</code> 指令的使用。</p>
<p>通过 <code class="highlighter-rouge">gensym</code> 指令可以直接返回一个临时文件句柄来使用。</p>
<h3 id="cb-">$cb->()</h3>
<p>在 SPEC 设计中,所有导出指令都采用回调的方式。在 Perl 中实现起来其实特别简单。像下面这样就好了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">sub </span><span class="nf">keyword</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$str</span><span class="p">,</span> <span class="nv">$cb</span> <span class="p">)</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$res</span> <span class="o">=</span> <span class="nv">do_some_func</span><span class="p">(</span><span class="nv">$str</span><span class="p">);</span>
<span class="nv">$cb</span><span class="o">-></span><span class="p">(</span><span class="nv">$res</span><span class="p">);</span>
<span class="p">}</span>
</code></pre>
</div>
<h3 id="selectortoxpath">selector_to_xpath</h3>
<p>之前一直有使用 <code class="highlighter-rouge">Mojo::UserAgent</code> 配合 <code class="highlighter-rouge">Mojo::DOM</code> 完成网页抓取工作,这次自己实践,参考的是另一个 <a href="https://metacpan.org/module/Web::Query">Web::Query</a> 模块。其中最关键的两步,第一是通过 <a href="https://metacpan.org/module/HTML::Selector::XPath">selector_to_xpath</a> 指令把选择器的写法转换成 XPath 语言;第二是通过 XPath 操作网页的 <a href="https://metacpan.org/module/HTML::TreeBuilder::XPath">HTML::Tree</a>。</p>
<p>不过 <code class="highlighter-rouge">Mojo</code> 里对象化的很完整,返回的数组和字符串都是对象,所以可以一直反复调用方法连接起来处理,写的会很爽。用 <code class="highlighter-rouge">Web::Query</code> 没有这个效果。</p>
<h3 id="filestat">File::stat</h3>
<p>stat 是 perl 默认的函数,不过返回的数组在 mode 和 time 方面可读性都不好,所以封装一下,提供更加可读的 0644 这样的 mode 格式,直接用 <code class="highlighter-rouge">sprintf</code> 就可以做到:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">sprintf</span><span class="p">(</span> <span class="s">"%04o"</span><span class="p">,</span> <span class="nv">$ret</span><span class="o">-></span><span class="nv">get</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="o">&</span> <span class="mo">07777</span> <span class="p">);</span>
</code></pre>
</div>
<h3 id="datetime">DateTime</h3>
<p>Perl 的 <a href="https://metacpan.org/module/DateTime">DateTime</a> 模块太重,CPAN 上其实也有很多人提交简化版的 DT,其实就是利用 <code class="highlighter-rouge">localtime</code>,<code class="highlighter-rouge">strftime</code> 和 <code class="highlighter-rouge">mktime</code> 几个默认函数做出来的对象调用。</p>
<h3 id="exporter">Exporter</h3>
<p><code class="highlighter-rouge">import</code> 和 <code class="highlighter-rouge">export_to_level</code> 都是 <code class="highlighter-rouge">Exporter</code> 模块的方法,所有继承自 <code class="highlighter-rouge">Exporter</code> 的模块可以用。比如下面示例,启用该模块,就相当于启用了 <code class="highlighter-rouge">strict</code>,<code class="highlighter-rouge">warnings</code>,<code class="highlighter-rouge">utf8</code> 和 Perl5.10 版的新特性,同时导出了 keywords 关键字。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nv">base</span> <span class="s">'Exporter'</span><span class="p">;</span>
<span class="k">our</span> <span class="nv">@EXPORT</span> <span class="o">=</span> <span class="sx">qw/keywords/</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">keywords</span> <span class="p">{</span> <span class="o">...</span> <span class="p">}</span>
<span class="k">sub </span><span class="nf">import</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$class</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="nv">strict</span><span class="o">-></span><span class="nb">import</span><span class="p">;</span>
<span class="nv">warnings</span><span class="o">-></span><span class="nb">import</span><span class="p">;</span>
<span class="nv">utf8</span><span class="o">-></span><span class="nb">import</span><span class="p">;</span>
<span class="nv">feature</span><span class="o">-></span><span class="nb">import</span><span class="p">(</span><span class="s">':5.10'</span><span class="p">);</span>
<span class="nn">Try::</span><span class="nv">Tiny</span><span class="o">-></span><span class="nb">import</span><span class="p">;</span>
<span class="nv">$class</span><span class="o">-></span><span class="nv">export_to_level</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nv">$class</span><span class="p">,</span> <span class="nv">@EXPORT</span><span class="p">);</span>
<span class="p">}</span>
</code></pre>
</div>
<h3 id="zip">zip</h3>
<p>多数组可以通过 <code class="highlighter-rouge">zip</code> 命令逐一对位融合到一起。这个在 <a href="https://metacpan.org/module/List::MoreUtils">List::MoreUtils</a> 中有,这次用 <code class="highlighter-rouge">NG::Array</code> 对象实现了一边,其原理是先记录每个数组的长度,然后以最长的那个数组为标杆,循环一遍即可。</p>
<h3 id="autobox">autobox</h3>
<p>CPAN 上 Rubyish、Perl6::<em>、Perl5i::</em> 等模块都利用了 <a href="https://metacpan.org/module/autobox">autobox</a> 实现完全的对象化。autobox 是一个库,本身不提供对象方法,而是要自己自己实现针对某个类型的对象方法后,通过 autobox 关联到 Perl 的数据类型上去。</p>
<p>比如想要实现一个 <code class="highlighter-rouge">"Hello World"->lc->words</code> 的语法,显然就是要针对 Perl 中的 STRING 数据类型实现 lc 和 words 两个方法。那么先实现一个自己的 string 对象:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">package</span> <span class="nn">your::</span><span class="nv">string</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">lc</span> <span class="p">{</span> <span class="nn">CORE::</span><span class="nv">lc</span> <span class="nv">$_</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="p">}</span>
<span class="k">sub </span><span class="nf">words</span> <span class="p">{</span> <span class="nn">CORE::</span><span class="nv">split</span> <span class="sr">/\s+/</span><span class="p">,</span> <span class="nv">$_</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="p">}</span>
<span class="mi">1</span><span class="p">;</span>
</code></pre>
</div>
<p>然后开始关联:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">package</span> <span class="nn">your::</span><span class="nv">autobox</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">base</span> <span class="sx">qw(autobox)</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">your::</span><span class="nv">string</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">import</span> <span class="p">{</span>
<span class="nb">shift</span><span class="o">-></span><span class="nn">SUPER::</span><span class="nv">import</span><span class="p">(</span>
<span class="nv">STRING</span> <span class="o">=></span> <span class="s">'your::string'</span><span class="p">,</span>
<span class="nv">@_</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="mi">1</span><span class="p">;</span>
</code></pre>
</div>
<p>最后在前面提到过的 <code class="highlighter-rouge">Exporter</code> 的 <code class="highlighter-rouge">import</code> 函数里加上一行:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nn">your::</span><span class="nv">autobox</span><span class="o">-></span><span class="nb">import</span><span class="p">;</span>
</code></pre>
</div>
<p>autobox 可以关联的数据类型还有很多,绝对是值得一看的模块。</p>
<h3 id="evalclassnew">eval(‘*’.$class.’::new’)</h3>
<p>实现 <code class="highlighter-rouge">def_class</code> 关键词的过程中学习颇多,首先是符号表。实现中完成模块代码几乎全靠符号表来绑定一个个函数和变量。像这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="o">*</span><span class="nv">t</span> <span class="o">=</span> <span class="nb">eval</span><span class="p">(</span><span class="s">'*'</span><span class="o">.</span><span class="nv">$class</span><span class="o">.</span><span class="s">'::ISA'</span><span class="p">);</span>
<span class="o">*</span><span class="nv">t</span> <span class="o">=</span> <span class="p">[</span><span class="nv">$parent</span><span class="p">];</span>
<span class="o">*</span><span class="nv">t</span> <span class="o">=</span> <span class="nb">eval</span><span class="p">(</span><span class="s">'*'</span><span class="o">.</span><span class="nv">$class</span><span class="o">.</span><span class="s">'::new'</span><span class="p">);</span>
<span class="o">*</span><span class="nv">t</span> <span class="o">=</span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$class</span><span class="p">,</span> <span class="nv">@args</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="nb">push</span> <span class="nv">@args</span><span class="p">,</span> <span class="s">''</span> <span class="k">if</span> <span class="nv">$#args</span> <span class="nv">%</span> <span class="nv">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$o</span> <span class="o">=</span> <span class="nb">bless</span> <span class="p">{</span><span class="nv">@args</span><span class="p">},</span> <span class="nb">ref</span> <span class="nv">$class</span> <span class="o">||</span> <span class="nv">$class</span><span class="p">;</span>
<span class="k">if</span><span class="p">(</span><span class="nb">defined</span> <span class="nv">$methods</span><span class="o">-></span><span class="p">{</span><span class="nv">build</span><span class="p">}){</span>
<span class="nv">$o</span><span class="o">-></span><span class="nv">build</span><span class="p">(</span><span class="nv">@args</span><span class="p">);</span>
<span class="p">}</span>
<span class="nv">$o</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
<p>不过这个实现有个问题,就是对象只能是基于哈希的引用,不能是数组的了。</p>
<h3 id="section">对象的元数据</h3>
<p>实现 <code class="highlighter-rouge">def_class</code> 的时候比 spec 多新增了一个默认属性叫meta,所有用 <code class="highlighter-rouge">def_class</code> 实现的类,会自动记录他们(包括他们的用 <code class="highlighter-rouge">def_class</code> 实现的父类)的属性和方法到meta属性里。</p>
<p>为此阅读了一下 <code class="highlighter-rouge">Moo</code> 和 <code class="highlighter-rouge">Moos</code> 的代码。<br />
<strong>原来他们都是把属性和方法也实现为类。然后再有 <code class="highlighter-rouge">*::Meta</code> 类来记录这些属性和方法的类。</strong></p>
<p>而 <code class="highlighter-rouge">Newbie::Gift</code> 计划中没打算把对象化搞得这么彻底,所以就只是存了一个 hash 到 默认 meta 属性里。</p>
<h3 id="lvalue">:lvalue</h3>
<p>对象除了方法还要有属性,<code class="highlighter-rouge">def_class</code> 里也有实现,同样是用符号表绑定的。</p>
<p>不过这里用到了 Perl5.10 的一个新东西,函数属性,这里绑定的不是普通变量而是函数,但是函数只会读写一个变量值,具体的说就是使用 <code class="highlighter-rouge">sub :lvalue {}</code> 定义。使用方法如下所示:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">my</span> <span class="nv">$val</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">canmod</span> <span class="p">:lvalue {</span>
<span class="c1"># return $val; this doesn't work, don't say "return"</span>
<span class="nv">$val</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">nomod</span> <span class="p">{</span>
<span class="nv">$val</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">canmod</span><span class="p">()</span> <span class="o">=</span> <span class="mi">5</span><span class="p">;</span> <span class="c1"># assigns to $val</span>
<span class="nv">nomod</span><span class="p">()</span> <span class="o">=</span> <span class="mi">5</span><span class="p">;</span> <span class="c1"># ERROR</span>
</code></pre>
</div>
<p>lvalue 的说明见 <code class="highlighter-rouge">perldoc perlsub</code> 文档。在这里还是个比较有趣的用法的,这个用法来自 <code class="highlighter-rouge">Newbie::Gift</code> 项目另一位参与者 <a href="https://github.com/fmpdceudy">fmpdceudy</a>。</p>
使用 Foreman 来监控统计 puppet 的 reports 信息
2013-04-16T00:00:00+08:00
devops
puppet
http://chenlinux.com/2013/04/16/install-foreman
<p>foreman 是社区比较推荐的一款 puppet 辅助工具。可以用来实现 ENC 控制,class 编写,Facts 变量统计和 reports 分析查询等等。</p>
<p>鉴于我一直以来都是用 gem 安装 puppet,所以这里也就没法通过 yum/apt 来安装 foreman,只能源码操作了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> git clone https://github.com/theforeman/foreman.git -b develop
<span class="nb">cd </span>foreman
bundle install --without postgresql mysql mysql2
cp config/settings.yaml.example config/settings.yaml
cp config/database.yml.example config/database.yml
<span class="nv">RAILS_ENV</span><span class="o">=</span>production bundle <span class="nb">exec </span>rake db:migrate
rake puppet:import:hosts_and_facts <span class="nv">RAILS_ENV</span><span class="o">=</span>production
./script/rails server -p 3333 -e production -d
</code></pre>
</div>
<p>然后就可以通过3333端口访问并查看刚才导入的 Facts 变量了,默认的用户名密码是 admin/changeme。</p>
<p>新版本的 foreman,必须使用 smart-proxy 才能接收 reports。所以还要继续安装:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> git clone git://github.com/theforeman/smart-proxy.git
<span class="nb">cd </span>smart-proxy
sed -i <span class="s1">'s/^#:puppet:.*/:puppet: true/'</span> config/settings.yml
./bin/smart-proxy.rb
</code></pre>
</div>
<p>foreman 提供了一个 <a href="https://raw.github.com/theforeman/puppet-foreman/master/templates/foreman-report.rb.erb">ruby 脚本</a>,用来扩充 puppet 的 reports 功能。下载放到对应的 <code class="highlighter-rouge">${GEM_PATH}/gems/puppet-${version}/lib/puppet/reports/</code> 下,然后修改其中的 <code class="highlighter-rouge">$foreman_url</code> 变量即可。</p>
<p>我们也可以在 puppet 自带的 http.rb 基础上稍微修改得到相同效果,总的来说,就是通过 POST 方法,提交 <code class="highlighter-rouge">report => self.to_yaml</code> 到 <code class="highlighter-rouge">$foreman_url/reports/create?format=yml</code> 就可以了。</p>
<p>然后在 foreman 页面上配置 smart-proxy 地址。注意这里有个小坑:__如果你填写的是域名,那么解析出来的 ip 还要被反解验证一次。__我当初为了 puppet master 迁移方便,给 master 配置了一个单独的域名,包括 <code class="highlighter-rouge">puppet cert</code> 生成证书时也特意指定用这个域名,但是默认的 hostname 其实是另一个域名的。于是在此悲剧了很久。。。</p>
<p>错误的现象是:采用 <code class="highlighter-rouge">puppet master</code> 启动时,功能一切正常;采用 <code class="highlighter-rouge">rackup</code> + Nginx 代理的方式启动时,默认的 store 功能正常,而采用 foreman 接收 reports 的话,可以在 <code class="highlighter-rouge">rackup</code> 的访问日志中看到 POST 200 的记录,<code class="highlighter-rouge">foreman</code> 里却没有接到请求。</p>
<p>目前还不清楚为什么两种不同方式启动 puppet 的 master 会对 smart-proxy 造成什么区别影响,但是修改 foreman 里配置的 smart-proxy 地址为默认 hostname 而不是单独的域名后,就成功了。</p>
<p>另外一个使用上的小问题。foreman 页面上的 Reports 标签的 <code class="highlighter-rouge"><a href=""></code> 属性默认是带搜索参数 eventful 的。也就是说优先展示的是有事件发生的日志,比如 failed,restart 等等;而不是直接以日期排序。</p>
Graphite 安装
2013-04-03T00:00:00+08:00
python
graphite
http://chenlinux.com/2013/04/03/install-graphite
<p>Graphite 是近来比较流行的类 rrd tool 系统。不过官网的安装文档真的很烂,特记录一下自己的步骤。</p>
<p>由于是事后追忆,同样不保证好用……</p>
<div class="highlighter-rouge"><pre class="highlight"><code>apt-get install python-pip libapache2-mod-wsgi subversion git
git clone https://github.com/graphite-project/graphite-web.git
git clone https://github.com/graphite-project/carbon.git
git clone https://github.com/graphite-project/whisper.git
<span class="c"># 这两个是直接通过 pip 安装的不顶用,只能另外下非标准的包安装</span>
git clone https://github.com/graphite-project/ceres.git
svn checkout http://django-tagging.googlecode.com/svn/trunk/ tagging-trunk
<span class="nb">cd </span>whisper
sudo python setup.py install
<span class="nb">cd</span> ../carbon
python setup.py install
<span class="nb">cd</span> ../graphite-web
python check-dependencies.py
<span class="c"># 很奇怪 python 居然不自动解决依赖,check 出来一个列表还得自己来</span>
apt-get install python-memcache python-txamqp python-rrdtool python-pyparsing python-django
python setup.py install
<span class="nb">cd</span> ../ceres
python setup.py install
<span class="nb">cd</span> ../tagging-trunk
python setup.py install
groupadd graphite
ln -s /opt/graphite/examples/example-graphite-vhost.conf /etc/apache2/conf.d/graphite.conf
<span class="c"># 默认的 run/wsgi 会在 /etc/apache2/ 目录下,权限有问题</span>
sed -i <span class="s1">'s!^\(WSGISocketPrefix\) \(run/wsgi\)$!\1 /var/\2$!'</span> /etc/apache2/conf.d/graphite.conf
chown -R www-data:graphite /opt/graphite/storage/
service apache2 restart
<span class="nb">cd</span> /opt/graphite/webapp/graphite
cp local_settings.py.example local_settings.py
<span class="c"># 默认的 database 配置是针对 python2.4 的,需要开启针对 python2.5 以上版本的配置:</span>
<span class="c"># DATABASES = {</span>
<span class="c"># 'default': {</span>
<span class="c"># 'NAME': '/opt/graphite/storage/graphite.db',</span>
<span class="c"># 'ENGINE': 'django.db.backends.sqlite3',</span>
<span class="c"># 'USER': '',</span>
<span class="c"># 'PASSWORD': '',</span>
<span class="c"># 'HOST': '',</span>
<span class="c"># 'PORT': ''</span>
<span class="c"># }</span>
<span class="c"># }</span>
sed -i <span class="s1">'167,176s/^#//'</span> local_settings.py
python manage.py syncdb
<span class="nb">cd</span> /opt/graphite/conf
rename <span class="s1">'s/.example//'</span> <span class="k">*</span>.example
<span class="nb">cd</span> /opt/graphite/
<span class="c"># 会监听 2003 端口</span>
./bin/carbon-cache.py start
<span class="c"># 通过 socket 发送本机的 loadavg 到 2003 端口</span>
python /opt/graphite/examples/example-client.py
</code></pre>
</div>
<p>效果如下:</p>
<p><img src="/images/uploads/graphite-auto-refresh.png" alt="graphite" /></p>
<p>还可以点击 plot 成下面这样,并且添加 event 以供查看:</p>
<p><img src="/images/uploads/graphite-graphlot.png" alt="graphlot" /></p>
用 Perl6 解析 puppet 的配置语法
2013-04-02T00:00:00+08:00
perl
puppet
perl6
http://chenlinux.com/2013/04/02/parse-puppet-dsl-using-perl6
<p>前段时间看到报道说,puppet 的作者本来是用 perl 完成的原型设计,后来改用的 ruby。所以我想,目前这个 puppet 的 DSL 设计,用 perl 来完成的话,应该如何做。</p>
<p>这里碰到一个问题,就是 puppet 中 <code class="highlighter-rouge">resource_type</code> 的 <code class="highlighter-rouge">title</code> 后面有个冒号,这事儿比较麻烦,不过这时候我突然想到了 Perl6 ,稍微翻了一下文档,发现这事用 Perl6 来实现很容易:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nv">v6</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">infix</span><span class="p">:<:>($a, %b){</span>
<span class="k">return</span> <span class="nv">$a</span><span class="p">,</span> <span class="nv">%b</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">sub </span><span class="nf">service</span><span class="p">(&service) {</span>
<span class="k">my</span> <span class="nv">@res</span> <span class="o">=</span> <span class="o">&</span><span class="nv">service</span><span class="o">.</span><span class="p">();</span>
<span class="nv">say</span> <span class="nv">@res</span><span class="o">.</span><span class="nb">shift</span> <span class="o">=></span> <span class="nv">@res</span><span class="o">.</span><span class="nv">hash</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">class</span> <span class="nn">nginx::</span><span class="nv">install</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$nginxparams</span> <span class="o">=</span> <span class="s">"nginx"</span><span class="p">;</span>
<span class="nv">service</span> <span class="p">{</span> <span class="s">"$nginxparams"</span><span class="p">:</span>
<span class="nv">conf</span> <span class="o">=></span> <span class="s">"#"</span><span class="p">,</span>
<span class="nv">source</span> <span class="o">=></span> <span class="s">"http"</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>运行结果如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nv">perl6</span> <span class="sr">/data/</span><span class="nv">perl6</span><span class="sr">/script/</span><span class="nv">puppet</span><span class="o">-</span><span class="nv">style</span><span class="o">.</span><span class="nv">pl</span>
<span class="s">"nginx"</span> <span class="o">=></span> <span class="p">{</span><span class="s">"conf"</span> <span class="o">=></span> <span class="s">"#"</span><span class="p">,</span> <span class="s">"source"</span> <span class="o">=></span> <span class="s">"http"</span><span class="p">}</span>
</code></pre>
</div>
<p>当然实际上 puppet 要复杂很多,这里其实更多是为了说明 Perl6 如何自定义操作符~</p>
用 Mojo 命令行抓取数据完成自动更新 rpm 构建
2013-04-01T00:00:00+08:00
perl
mojolicious
rpm
http://chenlinux.com/2013/04/01/use-mojo-commandline-for-rpmbuild
<p>我一直很喜欢 <code class="highlighter-rouge">Dancer</code> 里的 keyword 方式,所以很少使用 <code class="highlighter-rouge">Mojolicious</code> 框架来写网站,不过 <code class="highlighter-rouge">Mojo::UserAgent</code> 和 <code class="highlighter-rouge">Mojo::DOM</code> 在一起作为爬虫工具使用,真是太方便了。这两天需要自己打包 <code class="highlighter-rouge">tengine</code> ,考虑自动化因素,需要从 <code class="highlighter-rouge">tengine</code> 和 其他第三方模块的 <code class="highlighter-rouge">github</code> 托管网页上定期查询其更新,都是一行代码就搞定了。整个 <code class="highlighter-rouge">Build.PL</code> 如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/env perl</span>
<span class="k">use</span> <span class="nn">Modern::</span><span class="nv">Perl</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">IPC::</span><span class="nv">Run</span> <span class="sx">qw(run)</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">File::</span><span class="nv">Slurp</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">POSIX</span> <span class="sx">qw(strftime)</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Template</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">ojo</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@ModuleList</span> <span class="o">=</span> <span class="sx">qw(
renren/ngx_http_accounting_module
agentzh/echo-nginx-module
agentzh/chunkin-nginx-module
simpl/ngx_devel_kit
calio/form-input-nginx-module
chaoslawful/lua-nginx-module
renren/ngx_http_consistent_hash
)</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$TengineMD5</span> <span class="o">=</span> <span class="p">(</span><span class="nb">split</span><span class="p">(</span><span class="sr">/ /</span><span class="p">,</span> <span class="nv">g</span><span class="p">(</span><span class="s">"http://tengine.taobao.org/download_cn.html"</span><span class="p">)</span><span class="o">-></span><span class="nv">dom</span><span class="o">-></span><span class="nv">at</span><span class="p">(</span><span class="s">".one_col li span"</span><span class="p">)</span><span class="o">-></span><span class="nv">text</span><span class="p">))[</span><span class="o">-</span><span class="mi">1</span><span class="p">];</span>
<span class="nv">write_file</span><span class="p">(</span><span class="s">"md5.txt"</span><span class="p">,</span> <span class="s">"firstimetorun"</span><span class="p">)</span> <span class="k">unless</span> <span class="o">-</span><span class="nv">f</span> <span class="s">"md5.txt"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$TengineOldMD5</span> <span class="o">=</span> <span class="nv">read_file</span><span class="p">(</span> <span class="s">"md5.txt"</span> <span class="p">);</span>
<span class="nv">say</span> <span class="nv">$TengineOldMD5</span><span class="p">;</span>
<span class="nv">say</span> <span class="nv">$TengineMD5</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$TengineMD5</span> <span class="ow">ne</span> <span class="nv">$TengineOldMD5</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">gettarball</span><span class="p">(</span><span class="o">\</span><span class="nv">@ModuleList</span><span class="p">);</span>
<span class="nv">write_file</span><span class="p">(</span><span class="s">"md5.txt"</span><span class="p">,</span> <span class="nv">$TengineMD5</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">gettarball</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$ModuleList</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$TengineUrl</span> <span class="o">=</span> <span class="nv">g</span><span class="p">(</span><span class="s">"http://tengine.taobao.org/download_cn.html"</span><span class="p">)</span><span class="o">-></span><span class="nv">dom</span><span class="o">-></span><span class="nv">at</span><span class="p">(</span><span class="s">".one_col li a"</span><span class="p">)</span><span class="o">-></span><span class="p">{</span><span class="nv">href</span><span class="p">};</span>
<span class="k">my</span> <span class="nv">$TengineVersion</span> <span class="o">=</span> <span class="nv">$1</span> <span class="k">if</span> <span class="nv">$TengineUrl</span> <span class="o">=~</span> <span class="sr">m!download/tengine-(.*).tar.gz!</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$TengineRelease</span> <span class="o">=</span> <span class="nv">strftime</span><span class="p">(</span><span class="s">"%Y%m%d%H%M"</span><span class="p">,</span><span class="nb">localtime</span><span class="p">);</span>
<span class="nv">run</span><span class="p">(</span><span class="s">'wget'</span><span class="p">,</span> <span class="s">"http://tengine.taobao.org/${TengineUrl}"</span><span class="p">,</span> <span class="s">'-O'</span><span class="p">,</span> <span class="s">"SOURCES/tengine-${TengineVersion}.tar.gz"</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">@ModuleFile</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$i</span> <span class="o">=</span> <span class="mi">10</span><span class="p">;</span>
<span class="k">for</span> <span class="k">my</span> <span class="nv">$Module</span> <span class="p">(</span> <span class="nv">@</span><span class="p">{</span> <span class="nv">$ModuleList</span> <span class="p">}</span> <span class="p">)</span> <span class="p">{;</span>
<span class="k">my</span> <span class="nv">$GitUrl</span> <span class="o">=</span> <span class="s">"https://github.com/${Module}"</span><span class="p">;</span>
<span class="nv">say</span> <span class="nv">$GitUrl</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$GitCommit</span> <span class="o">=</span> <span class="nb">substr</span><span class="p">(</span><span class="nv">g</span><span class="p">(</span><span class="s">"${GitUrl}"</span><span class="p">)</span><span class="o">-></span><span class="nv">dom</span><span class="o">-></span><span class="nv">at</span><span class="p">(</span><span class="s">".sha"</span><span class="p">)</span><span class="o">-></span><span class="nv">text</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">7</span><span class="p">);</span>
<span class="p">(</span> <span class="k">my</span> <span class="nv">$StoreName</span> <span class="o">=</span> <span class="nv">$Module</span> <span class="p">)</span> <span class="o">=~</span> <span class="sr">s!/!-!</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$StoreFile</span> <span class="o">=</span> <span class="s">"${StoreName}-${GitCommit}.tar.gz"</span><span class="p">;</span>
<span class="nb">push</span> <span class="nv">@ModuleFile</span><span class="p">,</span> <span class="p">[</span> <span class="s">"Source${i}"</span> <span class="o">=></span> <span class="s">"${StoreName}-${GitCommit}"</span> <span class="p">];</span>
<span class="nv">run</span><span class="p">(</span><span class="s">'wget'</span><span class="p">,</span> <span class="s">"${GitUrl}/tarball/master"</span><span class="p">,</span> <span class="s">'-O'</span><span class="p">,</span> <span class="s">"SOURCES/$StoreFile"</span><span class="p">);</span>
<span class="nv">$i</span><span class="o">++</span><span class="p">;</span>
<span class="p">}</span>
<span class="nb">unlink</span><span class="p">(</span><span class="s">'SPECS/tengine.spec'</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$template</span> <span class="o">=</span> <span class="nv">Template</span><span class="o">-></span><span class="k">new</span><span class="p">;</span>
<span class="nv">$template</span><span class="o">-></span><span class="nv">process</span><span class="p">(</span><span class="s">"tengine.spec.tt"</span><span class="p">,</span> <span class="p">{</span>
<span class="nv">TengineVersion</span> <span class="o">=></span> <span class="nv">$TengineVersion</span><span class="p">,</span>
<span class="nv">TengineRelease</span> <span class="o">=></span> <span class="nv">$TengineRelease</span><span class="p">,</span>
<span class="nv">TengineAddons</span> <span class="o">=></span> <span class="o">\</span><span class="nv">@ModuleFile</span><span class="p">,</span>
<span class="p">},</span> <span class="s">"SPECS/tengine.spec"</span><span class="p">);</span>
<span class="nv">buildrpm</span><span class="p">(</span><span class="nv">$TengineVersion</span><span class="p">,</span> <span class="nv">$TengineRelease</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">buildrpm</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$TengineVersion</span><span class="p">,</span> <span class="nv">$TengineRelease</span> <span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$out</span><span class="p">,</span> <span class="nv">$err</span> <span class="p">);</span>
<span class="nv">run</span> <span class="p">[</span><span class="s">'rpmbuild'</span><span class="p">,</span> <span class="s">'-bb'</span><span class="p">,</span> <span class="s">'SPECS/tengine.spec'</span><span class="p">],</span> <span class="nb">undef</span><span class="p">,</span> <span class="o">\</span><span class="nv">$out</span><span class="p">,</span> <span class="nv">$err</span><span class="p">;</span>
<span class="nv">mail2author</span><span class="p">(</span><span class="nv">$err</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">mail2author</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$output</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$body</span> <span class="o">=</span> <span class="nv">$output</span> <span class="p">?</span> <span class="s">"Build Error: $output"</span> <span class="p">:</span> <span class="s">"Build OK"</span><span class="p">;</span>
<span class="nv">p</span><span class="p">(</span><span class="s">"http://email.notify.d.xiaonei.com/eml/tengine-build/chenlin.rao"</span> <span class="o">=></span> <span class="p">{</span> <span class="nv">DNT</span> <span class="o">=></span> <span class="mi">1</span> <span class="p">}</span> <span class="o">=></span> <span class="nv">$body</span><span class="p">);</span>
<span class="p">}</span>
</code></pre>
</div>
<p>直接 <code class="highlighter-rouge">g</code> 就是 GET 方法, <code class="highlighter-rouge">p</code> 就是 POST 方法。然后 <code class="highlighter-rouge">->dom->at()</code> 后采用类似 <code class="highlighter-rouge">jQuery</code> 的写法就可以直接定位,然后还可以用 <code class="highlighter-rouge">->text</code> 来获取内容,或者 <code class="highlighter-rouge">->{attr}</code> 来获取属性值。</p>
<p>顺带,今天刚知道原来 <code class="highlighter-rouge">Template</code> 模块也有 <code class="highlighter-rouge">filter</code> 可用。<code class="highlighter-rouge">tengine.spec.tt</code> 中就用了一个大写过滤:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Summary: a HTTP and reverse proxy server
Name: tengine
Version: <span class="o">[</span>% TengineVersion %]
Release: <span class="o">[</span>% TengineRelease %]
Source0: %<span class="o">{</span>name<span class="o">}</span>-%<span class="o">{</span>version<span class="o">}</span>.tar.gz
Source1: init.nginx
Source2: logrotate.nginx
Source3: nginx-renren-conf.tar.gz
<span class="gp">[% </span>FOREACH Module IN TengineAddons -%]
<span class="gp">[% </span>Module.0 %]: <span class="o">[</span>% Module.1 %].tar.gz
<span class="gp">[% </span>END %]
Group: System Environment/Daemons
License: BSD
BuildRoot: %<span class="o">{</span>_tmppath<span class="o">}</span>/%<span class="o">{</span>name<span class="o">}</span>-%<span class="o">{</span>version<span class="o">}</span>-%<span class="o">{</span>release<span class="o">}</span>
Requires: pcre,zlib,lua
BuildRequires: pcre-devel,zlib-devel,lua-devel
Requires<span class="o">(</span>post<span class="o">)</span>: chkconfig
Conflicts: nginx
%description
Nginx with modules: 1<span class="o">)</span> ngx_http_consistent_hash; 2<span class="o">)</span> ngx_http_accounting_module; 3<span class="o">)</span> agentzh-chunkin-nginx-module.
%prep
<span class="c">#%setup -q </span>
%setup -n tengine-%<span class="o">{</span>version<span class="o">}</span>
tar zxvf %<span class="o">{</span>SOURCE3<span class="o">}</span>
<span class="gp">[% </span>FOREACH Module IN TengineAddons -%]
tar zxvf %<span class="o">{[</span>% Module.0 FILTER upper %]<span class="o">}</span>
<span class="gp">[% </span>END %]
...;
</code></pre>
</div>
Haml 简介
2013-03-28T00:00:00+08:00
http://chenlinux.com/2013/03/28/intro-haml
<p>Haml 是 Ruby 社区的一种 HTML 标记语言,它利用强制缩进和类似 jQuery 属性标签的风格,简化书写 HTML 的工作。文档见:<a href="http://haml.info/docs.html">http://haml.info/docs.html</a>。</p>
<p>下面是一段官网上的快速入门,从标准的 erb 模板转变成 haml 模板:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o"><</span><span class="n">div</span> <span class="nb">id</span><span class="o">=</span><span class="s1">'content'</span><span class="o">></span>
<span class="o"><</span><span class="n">div</span> <span class="k">class</span><span class="o">=</span><span class="s1">'left column'</span><span class="o">></span>
<span class="o"><</span><span class="n">h2</span><span class="o">></span><span class="no">Welcome</span> <span class="n">to</span> <span class="n">our</span> <span class="n">site!</span><span class="o"><</span><span class="sr">/h2>
<p><%= print_infomation %></</span><span class="nb">p</span><span class="o">></span>
<span class="o"><</span><span class="sr">/div>
<div class='right' id='item<%= item.id %>'>
<%= render :partial => "item" %>
</</span><span class="n">div</span><span class="o">></span>
<span class="o"><</span><span class="sr">/div>
</span></code></pre>
</div>
<p>用 haml 只用这么写:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#content</span>
<span class="p">.</span><span class="nf">left</span><span class="p">.</span><span class="nf">column</span>
<span class="o">%</span><span class="n">h2</span> <span class="no">Welcome</span> <span class="n">to</span> <span class="n">our</span> <span class="n">site!</span>
<span class="o">%</span><span class="nb">p</span><span class="o">=</span> <span class="n">print_information</span>
<span class="p">.</span><span class="nf">right</span><span class="p">{</span><span class="ss">:id</span> <span class="o">=></span> <span class="s2">"item</span><span class="si">#{</span><span class="n">item</span><span class="p">.</span><span class="nf">id</span><span class="si">}</span><span class="s2">"</span><span class="p">}</span>
<span class="o">=</span> <span class="n">render</span> <span class="ss">:partial</span> <span class="o">=></span> <span class="s2">"sidebar"</span>
</code></pre>
</div>
<p>看起来相当 cool,回头在 CPAN 上一翻,原来 perl 社区也有 port 过来的 <a href="https://metacpan.org/module/Text::Haml">Text::Haml</a> 了。根据 perl 的特点有所改变,但是省键盘的特点依然在。</p>
<p>下面是一个例子:<br />
```perl<br />
use Text::Haml;<br />
my $haml = Text::Haml->new();<br />
my $hash = {<br />
title => ‘my title’,<br />
content => { line1 => “test”, line2 => “test2” }<br />
};<br />
print $haml->render_file(‘test.haml’, %$hash);</p>
<div class="highlighter-rouge"><pre class="highlight"><code>
`test.haml` 如下:
```perl
%html{ :xmlns => "http://www.w3.org/1999/xhtml", :lang => "zh"}
%head
%title= $title
%body
#content
.container
%strong= $title
- for my $line ( keys %$content ) {
.row-fluid= $content->{$line}
- }
</code></pre>
</div>
<p>生成的 HTML 内容如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nt"><html</span> <span class="na">xmlns=</span><span class="s">'http://www.w3.org/1999/xhtml'</span> <span class="na">lang=</span><span class="s">'zh'</span><span class="nt">></span>
<span class="nt"><head></span>
<span class="nt"><title></span>my title<span class="nt"></title></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><div</span> <span class="na">id=</span><span class="s">'content'</span><span class="nt">></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">'container'</span><span class="nt">></span>
<span class="nt"><strong></span>my title<span class="nt"></strong></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">'row-fluid'</span><span class="nt">></span>test<span class="nt"></div></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">'row-fluid'</span><span class="nt">></span>test2<span class="nt"></div></span>
<span class="nt"></div></span>
<span class="nt"></div></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</code></pre>
</div>
<p>Text::Haml 还提供了一个初始化参数 <code class="highlighter-rouge">vars_as_subs</code>,可以把变量变成同名函数,这样写起来就更像 ruby 了。不过目前只能是纯变量,复杂语句还是不行,所以好看不中用……</p>
<p>Text::Haml 向 Text::Xslate 学习,也提供了 <code class="highlighter-rouge">cache_dir</code>, <code class="highlighter-rouge">filter</code> 等等功能,所以性能和功能方面应该也不差。</p>
<p><a href="https://metacpan.org/module/Template::Tookit">Template::Tookit</a> 也有插件 <a href="https://metacpan.org/module/Template::Plugin::Haml">Template::Plugin::Haml</a> 可以参看。</p>
<h3 id="wrappertt">wrapper.tt</h3>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">!!!</span> <span class="mi">5</span>
<span class="nv">%html</span>
<span class="p">[</span><span class="nv">%</span> <span class="nv">content</span> <span class="nv">%</span><span class="err">]</span>
</code></pre>
</div>
<h3 id="hellott">hello.tt</h3>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">[</span><span class="nv">%</span><span class="err">-</span> <span class="nv">message</span><span class="o">=</span><span class="s">'Hello World'</span> <span class="nv">%</span><span class="err">]</span>
<span class="err">[%-</span> <span class="nv">USE</span> <span class="nv">Haml</span> <span class="o">-</span><span class="nv">%</span><span class="err">]</span>
<span class="err">[%-</span> <span class="nv">WRAPPER</span> <span class="nv">wrapper</span><span class="o">.</span><span class="nv">tt</span> <span class="o">|</span> <span class="nv">haml</span> <span class="o">-</span><span class="nv">%</span><span class="err">]</span>
<span class="err">[%-</span> <span class="nv">FILTER</span> <span class="nv">haml</span> <span class="o">-</span><span class="nv">%</span><span class="err">]</span>
<span class="err">%</span><span class="nv">head</span>
<span class="nv">%meta</span><span class="p">{:</span><span class="nv">charset</span> <span class="o">=></span> <span class="s">"utf-8"</span><span class="p">}</span>
<span class="nv">%title</span> <span class="nv">hello</span>
<span class="nv">%body</span>
<span class="nv">%p</span> <span class="p">[</span><span class="nv">%</span> <span class="nv">message</span> <span class="nv">%</span><span class="err">]</span>
<span class="err">%</span><span class="nv">ul</span>
<span class="p">[</span><span class="nv">%</span><span class="err">-</span> <span class="nv">total</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="nv">WHILE</span> <span class="nv">total</span> <span class="o"><</span> <span class="mi">5</span> <span class="nv">%</span><span class="err">]</span>
<span class="err">%</span><span class="nv">li</span> <span class="p">[</span><span class="nv">%</span> <span class="nv">total</span><span class="o">=</span><span class="nv">total</span><span class="o">+</span><span class="mi">1</span> <span class="nv">%</span><span class="err">][%</span> <span class="nv">total</span> <span class="nv">%</span><span class="err">]</span>
<span class="err">[%-</span> <span class="nv">END</span> <span class="o">-</span><span class="nv">%</span><span class="err">]</span>
<span class="err">[%-</span> <span class="nv">END</span> <span class="o">-</span><span class="nv">%</span><span class="err">]</span>
</code></pre>
</div>
<p>perl 三大 web 框架 Catalyst/Mojo/Dancer也都有对应的模板插件。</p>
用 Mod_Gearman 实现 Nagios 分布式
2013-03-27T00:00:00+08:00
monitor
nagios
gearman
http://chenlinux.com/2013/03/27/distributed-nagios-by-mod-gearman
<p>在 2011 年年底,我曾经连续写过四篇介绍 OMD 的文章。</p>
<ol>
<li><a href="http://chenlinux.com/2011/12/19/omd_intro_install_on_centos5/">http://chenlinux.com/2011/12/19/omd_intro_install_on_centos5/</a></li>
<li><a href="http://chenlinux.com/2011/12/27/conf_run_mod_gearman/">http://chenlinux.com/2011/12/27/conf_run_mod_gearman/</a></li>
<li><a href="http://chenlinux.com/2011/12/20/omd_configurations_basic/">http://chenlinux.com/2011/12/20/omd_configurations_basic/</a></li>
<li><a href="http://chenlinux.com/2011/12/20/shinken_discovery_runner/">http://chenlinux.com/2011/12/20/shinken_discovery_runner/</a></li>
</ol>
<p>不过之前都停留在代码观摩和安装文档的阶段。这几天刚好有点需求,真正测试了一下如何利用 mod_gearman 实现 分布式的 Nagios 监测集群。</p>
<p>OMD 的安装一如既往的简单,尤其是作为中控端,不需要讲究太多通用性,可以选择使用 ubuntu 系统,直接通过 deb 安装:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget http://omdistro.org/attachments/download/197/omd-0.56_0.wheezy_i386.deb
dpkg -i omd-0.56_0.wheezy_i386.deb
omd create cdn-monitor
su - cdn-monitor
omd start
</code></pre>
</div>
<p>这就已经启动了。</p>
<p>不过要使用 mod_gearman 的话,还需要通过 <code class="highlighter-rouge">omd config</code> 界面开启。</p>
<p>默认开启之后,是运行在本机多 worker 的 Load Balance 状态下。我们现在要做的是把worker拆分到其他机房去变成 Distributed 状态。</p>
<p><img src="/images/uploads/sample_distributed.png" alt="distributed" /></p>
<p>图上已经列出 server 和 worker 的主要配置不同。我们只需要照着这样改就可以了。</p>
<p>不过在作为纯 worker 端的机房服务器上,我们没有必要安装完整的 OMD 了,这厮安装包都有100MB大……</p>
<p><a href="http://mod-gearman.org/download/v1.4.2/">http://mod-gearman.org/download/v1.4.2/</a> 上提供了 mod_gearman 的独立安装包,我们只需要根据服务器发行版选择下载就可以,这里以 CentOS6 为例,相信现在这个也应该是服务器的主流。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget http://mod-gearman.org/download/v1.4.2/rhel6/x86_64/gearmand-0.25-1.rhel6.x86_64.rpm
wget http://mod-gearman.org/download/v1.4.2/rhel6/x86_64/mod_gearman-1.4.2-1.e.rhel6.x86_64.rpm
rpm -ivh gearmand-0.25-1.rhel6.x86_64.rpm mod_gearman-1.4.2-1.e.rhel6.x86_64.rpm
</code></pre>
</div>
<p>除了图中列出的几行关键配置以外,还有两个地方是需要修改的:</p>
<h3 id="gearmand-">gearmand 的监听</h3>
<p>OMD 安装的 gearmand 默认是监听在 127.0.0.1 上的,需要修改<code class="highlighter-rouge">/omd/sites/cdn-monitor/etc/mod-gearman/port.conf</code> 文件变成可以被其他机器访问的 IP 地址并重启。</p>
<p>同样 分布式的 <code class="highlighter-rouge">/etc/mod_gearman/mod_gearman_worker.conf</code> 里,也需要修改 server 配置并重启服务。</p>
<h3 id="encryption-">encryption 配置</h3>
<p>OMD 默认启用 encryption 并且会在 <code class="highlighter-rouge">/omd/sites/cdn/etc/mod-gearman/</code> 下生成 <code class="highlighter-rouge">secret.key</code> 文件。</p>
<p>但是 <code class="highlighter-rouge">mod_gearman</code> 默认开启 encryption ,却不可能知道中控端的密码,所以默认是在配置文件中指定的 <code class="highlighter-rouge">key=should_be_changed</code>。这里我们需要修改一致:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>scp nagios:/omd/sites/cdn/etc/mod-gearman/secret.key /etc/mod_gearman/
sed <span class="s1">'s!#keyfile.*!keyfile=/etc/mod_gearman/secret.key!'</span> /etc/mod_gearman/mod_gearman_worker.conf
service mod_gearman_worker restart
</code></pre>
</div>
<p>事情还没完。这时候你会在 webUI 上看到分配给这个 worker 的检测全部报错,退出码 127。具体内容是:”/omd/sites/cdn-monitor/lib/nagios/plugins/check_http do not exists”之类的话。</p>
<p>因为,在 OMD 上,commands.cfg 上,配置的 <code class="highlighter-rouge">$USER1$/check_http</code> 替换为具体路径后,直接 <code class="highlighter-rouge">add_task</code> 到 gearmand 里,所以 worker 上收到 command 并执行也就是这样的了。目前还没有发现可以在 worker 端替换 commands 字符串的简单办法。所以,我们还得自己创建一个软链接:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>mkdir -p /omd/sites/cdn-monitor/lib/nagios/
yum install -y nagios-plugins-all --enablerepo<span class="o">=</span>epel
ln -s /usr/lib64/nagios/plugins /omd/sites/cdn-monitor/lib/nagios/plugins
</code></pre>
</div>
<p>OK,现在这个机房(即nagios配置中的hostgroup)的监测任务,就都分发给本机房的 worker 来进行了。比如 <code class="highlighter-rouge">check_http</code> 任务,可以看到原先跨机房访问带来的几十毫秒的延时,都变成了一两毫秒。</p>
logrotate 配置文件强制为 0644 属性
2013-03-18T00:00:00+08:00
linux
C
http://chenlinux.com/2013/03/18/logrotate-configuration-files-mode
<p>在一次包更新后,发现 Nginx 服务器的每晚日志切割不再进行了。找遍了各种地方,最后在一次偶然的<code class="highlighter-rouge">ls -l</code>中发现:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># ll /etc/logrotate.d/</span>
total 64
-rw-r--r-- 1 root root 326 2012-08-04 06:08 apache2
-rw-r--r-- 1 root root 84 2009-02-08 05:18 apt
-rw-r--r-- 1 root root 79 2008-12-05 17:15 aptitude
-rw-r--r-- 1 root root 330 2008-03-08 05:36 atop
-rw-r--r-- 1 root root 232 2011-11-10 14:33 dpkg
-rw-r--r-- 1 root root 267 2013-01-31 13:20 foreman-proxy
-rw-r--r-- 1 root root 151 2007-09-29 19:23 iptraf
-rw-r--r-- 1 root root 880 2012-10-29 17:10 mysql-server
-rwxr-xr-x 1 root root 356 2012-08-05 00:17 nginx
-rw-r--r-- 1 root root 1061 2008-03-08 05:36 psaccs_atop
-rw-r--r-- 1 root root 512 2008-03-08 05:36 psaccu_atop
-rw-r--r-- 1 root root 260 2012-06-23 00:52 rabbitmq-server
-rw-r--r-- 1 root root 126 2012-06-09 00:22 redis-server
-rw-r--r-- 1 root root 515 2012-09-27 02:40 rsyslog
-rw-r--r-- 1 root root 285 2008-11-18 21:20 stunnel4
</code></pre>
</div>
<p>这里的nginx多了可执行权限。于是我尝试性的执行了<code class="highlighter-rouge">chmod -x nginx</code>;结果居然真的恢复了。</p>
<p>这事儿说起来蛮奇怪了。于是去 <a href="https://fedorahosted.org/logrotate">https://fedorahosted.org/logrotate</a> 找来 logrotate 的源码看,结果在<code class="highlighter-rouge">logrotate-3.8.3/config.c</code> 里发现这么一段:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="mi">661</span> <span class="nf">if</span> <span class="p">((</span><span class="n">sb</span><span class="p">.</span><span class="n">st_mode</span> <span class="o">&</span> <span class="mo">07533</span><span class="p">)</span> <span class="o">!=</span> <span class="mo">0400</span><span class="p">)</span> <span class="p">{</span>
<span class="mi">662</span> <span class="n">message</span><span class="p">(</span><span class="n">MESS_DEBUG</span><span class="p">,</span>
<span class="mi">663</span> <span class="s">"Ignoring %s because of bad file mode.</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span>
<span class="mi">664</span> <span class="n">configFile</span><span class="p">);</span>
<span class="mi">665</span> <span class="n">close</span><span class="p">(</span><span class="n">fd</span><span class="p">);</span>
<span class="mi">666</span> <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="mi">667</span> <span class="p">}</span>
</code></pre>
</div>
<p>只有文件权限是 0644 的时候,配置文件才会被读取!0755 的与结果是 0511,不等于 0400。相关 <code class="highlighter-rouge">st_mode</code> 的内容可以通过 <code class="highlighter-rouge">man 2 stat</code> 查看。</p>
<p>可以写一小段 perl 代码来验证:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/perl</span>
<span class="k">my</span> <span class="nv">$mode</span> <span class="o">=</span> <span class="p">(</span><span class="nb">stat</span><span class="p">(</span><span class="nv">$ARGV</span><span class="p">[</span><span class="mi">0</span><span class="p">]))[</span><span class="mi">2</span><span class="p">];</span>
<span class="nb">printf</span> <span class="s">"Permissions are %04o\n"</span><span class="p">,</span> <span class="nv">$mode</span> <span class="o">&</span> <span class="mo">07533</span><span class="p">;</span>
</code></pre>
</div>
<p>在 <a href="https://fedorahosted.org/logrotate/browser/tags/r3-8-3/CHANGES">ChangeLog</a> 里,看到如下一段话:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>2.1 -> 2.2:
- ignore nonnormal files when reading config files from a directory
- (these were suggested and originally implemented by
Henning Schmiedehausen)
</code></pre>
</div>
<p>不过比较早了,就懒得从历史堆里再翻为什么当初会有这么个提议了…………</p>
Puppet 自定义 Provider
2013-03-15T00:00:00+08:00
devops
puppet
ruby
http://chenlinux.com/2013/03/15/puppet-provider-development
<p>Puppet 默认提供了相当多的资源类型,不过我们还可以更进一步的扩展这个庞大的阵营。比如在 <code class="highlighter-rouge">package</code> 类型的资源里,我们看到 puppet 除了系统级别的<code class="highlighter-rouge">yum</code>,<code class="highlighter-rouge">apt</code>之类意外,还提供了 <code class="highlighter-rouge">gem</code>,<code class="highlighter-rouge">pip</code> 来管理 ruby 和 python 的 package。那么很自然的,我们就可以进一步扩充 <code class="highlighter-rouge">package</code> 来管理 perl 的 package 。只需要新加一个 provider 就可以了。</p>
<p>关于 provider 开发的原理说明,见 <a href="http://docs.puppetlabs.com/guides/provider_development.html">http://docs.puppetlabs.com/guides/provider_development.html</a>。</p>
<p>下面是 <code class="highlighter-rouge">/etc/puppet/modules/production/myclass/lib/puppet/provider/package/cpan.rb</code> 的内容,他会被 puppet 以 <code class="highlighter-rouge">pluginsync</code> 的方式下发。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1"># 加载父类,这里是扩展 package 功能</span>
<span class="nb">require</span> <span class="s1">'puppet/provider/package'</span>
<span class="no">Puppet</span><span class="o">::</span><span class="no">Type</span><span class="p">.</span><span class="nf">type</span><span class="p">(</span><span class="ss">:package</span><span class="p">).</span><span class="nf">provide</span> <span class="ss">:cpan</span><span class="p">,</span> <span class="ss">:parent</span> <span class="o">=></span> <span class="no">Puppet</span><span class="o">::</span><span class="no">Provider</span><span class="o">::</span><span class="no">Package</span> <span class="k">do</span>
<span class="n">desc</span> <span class="s2">"CPAN modules support. You can pass any `source` which `cpanm` support,
like URL, git repos and local tar.gz. If source is not present at all,
the module will be installed from the default CPAN source.
You must install App::cpanminus, App::pmodinfo, App::pmuninstall before."</span>
<span class="n">has_feature</span> <span class="ss">:versionable</span>
<span class="c1"># 下面这个是 Puppet::Provider 提供的私有方法,用来指定类内部适用的系统命令</span>
<span class="c1"># puppet agent 会通过对这个的运行测试来确认该 provider 是否适用于本机</span>
<span class="c1"># 所以在使用这个 provider 之前,要先通过其他方式在 node 上安装好这三个命令</span>
<span class="n">commands</span> <span class="ss">:cpanmcmd</span> <span class="o">=></span> <span class="s2">"cpanm"</span>
<span class="n">commands</span> <span class="ss">:pmodinfocmd</span> <span class="o">=></span> <span class="s2">"pmodinfo"</span>
<span class="n">commands</span> <span class="ss">:pmuninstallcmd</span> <span class="o">=></span> <span class="s2">"pm-uninstall"</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">pmodlist</span><span class="p">(</span><span class="n">options</span><span class="p">)</span>
<span class="n">pmodlist_command</span> <span class="o">=</span> <span class="p">[</span><span class="n">command</span><span class="p">(</span><span class="ss">:pmodinfocmd</span><span class="p">),]</span>
<span class="k">if</span> <span class="n">options</span><span class="p">[</span><span class="ss">:local</span><span class="p">]</span>
<span class="n">pmodlist_command</span> <span class="o"><<</span> <span class="s2">"-l"</span>
<span class="k">else</span>
<span class="n">pmodlist_command</span> <span class="o"><<</span> <span class="s2">"-c"</span>
<span class="k">end</span>
<span class="k">if</span> <span class="nb">name</span> <span class="o">=</span> <span class="n">options</span><span class="p">[</span><span class="ss">:justme</span><span class="p">]</span>
<span class="n">pmodlist_command</span> <span class="o"><<</span> <span class="nb">name</span>
<span class="c1"># execute 是 Puppet::Util::Execution 提供的方法,接受数组传入,输出标准输出结果字符串</span>
<span class="n">list</span> <span class="o">=</span> <span class="p">[</span><span class="n">execute</span><span class="p">(</span><span class="n">pmodlist_command</span><span class="p">)].</span><span class="nf">map</span> <span class="p">{</span><span class="o">|</span><span class="n">set</span><span class="o">|</span> <span class="n">pmodsplit</span><span class="p">(</span><span class="n">set</span><span class="p">)</span> <span class="p">}.</span><span class="nf">reject</span> <span class="p">{</span><span class="o">|</span><span class="n">x</span><span class="o">|</span> <span class="n">x</span><span class="p">.</span><span class="nf">nil?</span> <span class="p">}</span>
<span class="k">else</span>
<span class="n">list</span> <span class="o">=</span> <span class="n">execute</span><span class="p">(</span><span class="n">pmodlist_command</span><span class="p">).</span><span class="nf">lines</span><span class="p">.</span><span class="nf">map</span> <span class="p">{</span><span class="o">|</span><span class="n">set</span><span class="o">|</span> <span class="n">pmodsplit</span><span class="p">(</span><span class="n">set</span><span class="p">)</span> <span class="p">}.</span><span class="nf">reject</span> <span class="p">{</span><span class="o">|</span><span class="n">x</span><span class="o">|</span> <span class="n">x</span><span class="p">.</span><span class="nf">nil?</span> <span class="p">}</span>
<span class="k">end</span>
<span class="k">if</span> <span class="nb">name</span> <span class="o">=</span> <span class="n">options</span><span class="p">[</span><span class="ss">:justme</span><span class="p">]</span>
<span class="k">return</span> <span class="n">list</span><span class="p">.</span><span class="nf">shift</span>
<span class="k">else</span>
<span class="k">return</span> <span class="n">list</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">pmodsplit</span><span class="p">(</span><span class="n">desc</span><span class="p">)</span>
<span class="k">if</span> <span class="n">desc</span> <span class="o">=~</span> <span class="sr">/^(\S+) version is (.+)\.(\n Last cpan version: (.+))?/</span>
<span class="nb">name</span> <span class="o">=</span> <span class="vg">$1</span>
<span class="c1"># 整个rb是从gem.rb复制过来的,gem list -r所有版本列成一行,split成一个数组</span>
<span class="c1"># 这里为了改动少点,就照样做成数组</span>
<span class="n">versions</span> <span class="o">=</span> <span class="p">[</span><span class="vg">$2</span><span class="p">]</span>
<span class="k">if</span> <span class="n">latest_version</span> <span class="o">=</span> <span class="vg">$3</span>
<span class="n">versions</span><span class="p">.</span><span class="nf">unshift</span><span class="p">(</span><span class="vg">$4</span><span class="p">)</span>
<span class="k">end</span>
<span class="p">{</span>
<span class="ss">:name</span> <span class="o">=></span> <span class="nb">name</span><span class="p">,</span>
<span class="ss">:ensure</span> <span class="o">=></span> <span class="n">versions</span><span class="p">,</span>
<span class="ss">:provider</span> <span class="o">=></span> <span class="ss">:cpan</span>
<span class="p">}</span>
<span class="k">else</span>
<span class="no">Puppet</span><span class="p">.</span><span class="nf">warning</span> <span class="s2">"Could not match </span><span class="si">#{</span><span class="n">desc</span><span class="si">}</span><span class="s2">"</span> <span class="k">unless</span> <span class="n">desc</span><span class="p">.</span><span class="nf">chomp</span><span class="p">.</span><span class="nf">empty?</span>
<span class="kp">nil</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># 这个 instances 方法是 provider 必须提供,在package里就是本地模块的列表</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">instances</span><span class="p">(</span><span class="n">justme</span> <span class="o">=</span> <span class="kp">false</span><span class="p">)</span>
<span class="n">pmodlist</span><span class="p">(</span><span class="ss">:local</span> <span class="o">=></span> <span class="kp">true</span><span class="p">).</span><span class="nf">collect</span> <span class="k">do</span> <span class="o">|</span><span class="nb">hash</span><span class="o">|</span>
<span class="kp">new</span><span class="p">(</span><span class="nb">hash</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># 往下的方法都是 package 要求提供的</span>
<span class="k">def</span> <span class="nf">install</span><span class="p">(</span><span class="n">useversion</span> <span class="o">=</span> <span class="kp">true</span><span class="p">)</span>
<span class="n">command</span> <span class="o">=</span> <span class="p">[</span><span class="n">command</span><span class="p">(</span><span class="ss">:cpanmcmd</span><span class="p">)]</span>
<span class="c1"># cpanm 指定安装版本的命令格式是这样: cpanm Dancer@1.000</span>
<span class="n">resource</span><span class="p">[</span><span class="ss">:name</span><span class="p">]</span> <span class="o">+=</span> <span class="s1">'@'</span> <span class="o">+</span> <span class="n">resource</span><span class="p">[</span><span class="ss">:ensure</span><span class="p">]</span> <span class="k">if</span> <span class="p">(</span><span class="o">!</span> <span class="n">resource</span><span class="p">[</span><span class="ss">:ensure</span><span class="p">].</span><span class="nf">is_a?</span> <span class="no">Symbol</span><span class="p">)</span> <span class="n">and</span> <span class="n">useversion</span>
<span class="n">command</span> <span class="o"><<</span> <span class="n">resource</span><span class="p">[</span><span class="ss">:name</span><span class="p">]</span>
<span class="n">output</span> <span class="o">=</span> <span class="n">execute</span><span class="p">(</span><span class="n">command</span><span class="p">)</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">fail</span> <span class="s2">"Could not install: </span><span class="si">#{</span><span class="n">output</span><span class="p">.</span><span class="nf">chomp</span><span class="si">}</span><span class="s2">"</span> <span class="k">if</span> <span class="n">output</span><span class="p">.</span><span class="nf">include?</span><span class="p">(</span><span class="s2">"failed"</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">latest</span>
<span class="n">pmodinfo_options</span> <span class="o">=</span> <span class="p">{</span><span class="ss">:justme</span> <span class="o">=></span> <span class="n">resource</span><span class="p">[</span><span class="ss">:name</span><span class="p">]}</span>
<span class="nb">hash</span> <span class="o">=</span> <span class="nb">self</span><span class="p">.</span><span class="nf">class</span><span class="p">.</span><span class="nf">pmodlist</span><span class="p">(</span><span class="n">pmodlist_options</span><span class="p">)</span>
<span class="c1"># 这里就是前面要用数组的原因了</span>
<span class="nb">hash</span><span class="p">[</span><span class="ss">:ensure</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span>
<span class="k">end</span>
<span class="c1"># 请求本地是否存在具体某个包</span>
<span class="k">def</span> <span class="nf">query</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">class</span><span class="p">.</span><span class="nf">pmodlist</span><span class="p">(</span><span class="ss">:justme</span> <span class="o">=></span> <span class="n">resource</span><span class="p">[</span><span class="ss">:name</span><span class="p">],</span> <span class="ss">:local</span> <span class="o">=></span> <span class="kp">true</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">uninstall</span>
<span class="n">pmuninstallcmd</span> <span class="n">resource</span><span class="p">[</span><span class="ss">:name</span><span class="p">]</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">update</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">install</span><span class="p">(</span><span class="kp">false</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p>在一台没有安装 cpanm 等命令的主机上运行 <code class="highlighter-rouge">puppet agent --debug</code>,可以看到这么一行输出:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>debug: Puppet::Type::Package::ProviderCpan: file cpanm does not exist
</code></pre>
</div>
极光推送demo
2013-03-14T00:00:00+08:00
monitor
android
http://chenlinux.com/2013/03/14/JPush-example
<p>之前已经陆续写过很多种告警的方式。今天再稍微试验一种更新潮一些的 —— 手机推送通知。原先我的想法是移植 HTML5 的 websocket + notification 页面到手机上。但是发现手机上的浏览器都还没有 notification 功能。即便是用 PhoneGap 包装 HTML5 应用,PhoneGap 的 notification API 也不是我想象中的状态栏通知,而是类似 js 的 alert 对话框。</p>
<p>不过这个时候我发现了极光推送。嗯,本来蛮有挑战的事情顿时变成了十分钟内解决的小菜:</p>
<p>整个过程如下:</p>
<h3 id="section">注册帐号</h3>
<p>官网地址: <a href="http://jpush.cn">http://jpush.cn</a></p>
<h3 id="section-1">新建应用</h3>
<p>都是纯页面操作,填写应用名称而已。</p>
<h3 id="example-">下载 example 包</h3>
<p>在应用详情里有下载链接。</p>
<h3 id="adt-eclipse--example">用 adt eclipse 打开 example</h3>
<p>adt eclipse 直接从 android 官网下载 adt-bundle-linux-x86-20130219.tar.gz 解压即可运行。更多配置见 android 官网说明。</p>
<p>然后在 File 菜单栏选择 new -> android application project 就可以新建自己的项目。然后创建自己的 workspace,把下好的 JPush example 包解压倒入workspace,然后就可以 run 了。</p>
<p>不过这里 run 会启动一个 android 虚拟机,很可能是连不上网的,原因似乎是 android vm 默认是 10.0.0.0 网段。</p>
<p>其实这时候我们会在 <code class="highlighter-rouge">workspace/push-example/bin/</code> 下发现一个 <code class="highlighter-rouge">push-example.apk</code> 文件。复制出来,通过豌豆荚或者别的什么工具直接装进自己手机就可以运行了。</p>
<h3 id="section-2">测试页面发送通知</h3>
<p>在极光的 portal 页面 <a href="http://www.jpush.cn/apps/${your app key}/notification">http://www.jpush.cn/apps/${your app key}/notification</a> 上可以直接提交通知内容。然后你就可以在手机状态栏通知上看到啦!</p>
<h3 id="section-3">测试命令行发送通知</h3>
<p>文档见<a href="http://docs.jpush.cn/pages/viewpage.action?pageId=2621796">http://docs.jpush.cn/pages/viewpage.action?pageId=2621796</a>。</p>
<p>比如通过简易通知推送接口发送如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="c">#!/bin/sh</span>
<span class="c">#下面两个是你新建应用后就分配的</span>
<span class="nv">APP_KEY</span><span class="o">=</span><span class="nv">$1</span>
<span class="nv">API_MasterSecret</span><span class="o">=</span><span class="nv">$2</span>
<span class="c">#自赠序列号,这个最好是通过mysql的auto_increment管理</span>
<span class="nv">sendno</span><span class="o">=</span>2
<span class="c">#这里有1,2,3,4,分别对应对指定IMEI/tag/alias/all的用户推送</span>
<span class="nv">receiver_type</span><span class="o">=</span>4
<span class="nv">verification_code</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> -ne <span class="s2">"</span><span class="nv">$sendno$receiver_type$API_MasterSecret</span><span class="s2">"</span> | md5sum | awk <span class="s1">'{print $1}'</span><span class="sb">`</span>
<span class="c">#platform包括android,ios等等,可以用逗号分开写多个</span>
curl http://api.jpush.cn:8800/sendmsg/v2/notification -d <span class="s2">"sendno=</span><span class="k">${</span><span class="nv">sendno</span><span class="k">}</span><span class="s2">&app_key=</span><span class="k">${</span><span class="nv">APP_KEY</span><span class="k">}</span><span class="s2">&receiver_type=</span><span class="k">${</span><span class="nv">receiver_type</span><span class="k">}</span><span class="s2">&platform=android&txt=123&verification_code=</span><span class="k">${</span><span class="nv">verification_code</span><span class="k">}</span><span class="s2">"</span>
</code></pre>
</div>
<p>然后收到如下响应:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="o">{</span><span class="s2">"sendno"</span> :<span class="s2">"2"</span>, <span class="s2">"errcode"</span>:0, <span class="s2">"errmsg"</span>:<span class="s2">"Succeed"</span><span class="o">}</span>
</code></pre>
</div>
<p>手机也同时响起~成功。</p>
Nginx 万兆网络环境测试
2013-02-25T00:00:00+08:00
testing
nginx
perl
apachebench
http://chenlinux.com/2013/02/25/nginx-testing-10Gibps
<ul id="markdown-toc">
<li><a href="#section" id="markdown-toc-section">测试目标</a></li>
<li><a href="#section-1" id="markdown-toc-section-1">测试设备</a></li>
<li><a href="#section-2" id="markdown-toc-section-2">测试环境说明</a></li>
<li><a href="#section-3" id="markdown-toc-section-3">特别注意事项</a> <ul>
<li><a href="#keepalive" id="markdown-toc-keepalive">Keepalive</a></li>
<li><a href="#rpspps" id="markdown-toc-rpspps">RPS、PPS与流量吞吐率</a></li>
<li><a href="#section-4" id="markdown-toc-section-4">压缩</a></li>
<li><a href="#section-5" id="markdown-toc-section-5">规则集</a></li>
<li><a href="#section-6" id="markdown-toc-section-6">实际网络情况</a></li>
</ul>
</li>
<li><a href="#section-7" id="markdown-toc-section-7">测试项目</a> <ul>
<li><a href="#section-8" id="markdown-toc-section-8">测试一</a></li>
<li><a href="#section-9" id="markdown-toc-section-9">测试二</a></li>
<li><a href="#section-10" id="markdown-toc-section-10">测试三</a></li>
<li><a href="#section-11" id="markdown-toc-section-11">测试四</a></li>
<li><a href="#section-12" id="markdown-toc-section-12">测试五</a></li>
<li><a href="#section-13" id="markdown-toc-section-13">测试六</a></li>
<li><a href="#section-14" id="markdown-toc-section-14">测试七</a></li>
</ul>
</li>
<li><a href="#section-15" id="markdown-toc-section-15">术语及缩写说明</a></li>
<li><a href="#section-16" id="markdown-toc-section-16">附注</a></li>
</ul>
<h1 id="section">测试目标</h1>
<p>本次测试的目标是将nginx作为7层负载均衡软件,应用于万兆环境下,获得</p>
<ul>
<li>
<p>极限性能</p>
</li>
<li>
<p>在服务质量保证约束条件下的极限性能</p>
</li>
<li>
<p>提升极限性能的方法</p>
</li>
<li>
<p>在万兆环境下部署的一般方法和特殊注意事项</p>
</li>
</ul>
<p>同时,由于nginx还作为静态页面的提供者,附带还进行获得nginx作为静态文件服务器的</p>
<ul>
<li>
<p>极限性能</p>
</li>
<li>
<p>在服务质量保证约束条件下的极限性能</p>
</li>
<li>
<p>提升极限性能的方法</p>
</li>
<li>
<p>在万兆环境下部署的一般方法和特殊注意事项</p>
</li>
</ul>
<p>操作系统是一个很大的影响因素,在测试开始,会对操作系统版本进行选择,以确定哪个是更合适的平台。</p>
<h1 id="section-1">测试设备</h1>
<p>测试设备如下</p>
<ul>
<li>
<p>万兆网卡X520-DA2</p>
</li>
<li>
<p>万兆交换机DCS-7124SX-F</p>
</li>
<li>
<p>万兆DA线</p>
</li>
<li>
<p>SNB服务器</p>
</li>
<li>
<p>WSM服务器</p>
</li>
</ul>
<h1 id="section-2">测试环境说明</h1>
<p>所有测试服务器连接在同一个万兆交换机下,处于同一个网段。</p>
<p>服务器划分为</p>
<ul>
<li>
<p>服务器(以S简称):提供静态或动态页面,次要测试对象</p>
</li>
<li>
<p>7层负载均衡(以LB简称):提供负载均衡功能,主要测试对象</p>
</li>
<li>
<p>客户端(以C简称):提供测试压力</p>
</li>
</ul>
<h1 id="section-3">特别注意事项</h1>
<h2 id="keepalive">Keepalive</h2>
<p>由于LB和S数量少,高RPS情况下,LB如果采用短连接方式连接S,LB的outgoing port会很快用尽。</p>
<p>解决的方法有两种:</p>
<ol>
<li>
<p>S采用多IP配置,模拟众多S的情况,减小每个S所要消耗的LB outgoing port;</p>
</li>
<li>
<p>LB与S之间采用keepalive连接</p>
</li>
</ol>
<p>短连接的方式下,CPU会有相当一部分消耗在三次握手上,这主要是操作系统开销(小包处理),以及nginx初始化会话上下文的开销。</p>
<h2 id="rpspps">RPS、PPS与流量吞吐率</h2>
<p>极限测试下,PPS考验服务器的CPU能力;而HTTP协议请求头的处理也是CPU密集型任务。</p>
<p>相同流量吞吐率下</p>
<ul>
<li>
<p>较小的响应意味着较高的RPS</p>
</li>
<li>
<p>当响应小于一个最大TCP报文长度时,响应越小,也意味着PPS越高</p>
</li>
</ul>
<p>极限测试中,需要测试以下三种情形</p>
<ul>
<li>
<p>全部是大报文。可以通过构造响应为一个最大TCP报文长度实现</p>
</li>
<li>
<p>全部是小报文。可以通过构造响应为最小HTTP响应实现</p>
</li>
<li>
<p>混合报文。可以通过构造响应为一个或两个TCP报文,一个是最大TCP报文长度,另外一个的长度可以控制,来满足特定的混合比例</p>
</li>
</ul>
<p>流量吞吐率是一个表观指标,但我们更关心的是在带宽足够的条件下,RPS指标(的范围)。</p>
<h2 id="section-4">压缩</h2>
<p>在我们的实际使用中,LB是不负责压缩的。但测试中需要测试压缩对RPS的影响。需要考察压缩方面的优化方法。</p>
<h2 id="section-5">规则集</h2>
<p>负载均衡规则集大小对性能会有影响。由于规则集处理开销正比于RPS,测试中应对较大的规则集进行测试以找出影响大小和可能的优化措施。</p>
<h2 id="section-6">实际网络情况</h2>
<p>一般而言,极限性能是最理想情况。在本次测试中,我们还需要测试接近实际网络情况下的极限性能,这可以通过引入丢包率、延时来模拟。</p>
<h1 id="section-7">测试项目</h1>
<h2 id="section-8">测试一</h2>
<p>测试目标:了解nginx的带宽满载的最小文件大小 (web服务器模式),确定之后测试的文件大小上限</p>
<p>测试工具:ab</p>
<p>测试方法:</p>
<ol>
<li>
<p>S上Nginx做简单配置,使其成为web server;</p>
</li>
<li>
<p>在C1上使用 <测试工具> 对S测试10000字节文件,每次减小1000字节,至带宽和RPS均为最高为止。</测试工具></p>
</li>
<li>
<p>根据第二步测试情况调整文件大小,以逼近S的带宽恰好满载且CPU占用率最高时的情况,并记录最终的阈值大小和RPS/PPS指标。</p>
</li>
</ol>
<p>测试结果:</p>
<table>
<thead>
<tr>
<th><strong>文件大小(Bytes)</strong></th>
<th><strong>CPU idle(%)</strong></th>
<th><strong>带宽(MBps)</strong></th>
<th style="text-align: right"><strong>PPS</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td>10K</td>
<td>65</td>
<td>1140</td>
<td style="text-align: right">606K</td>
</tr>
<tr>
<td>3K</td>
<td>12</td>
<td>1135</td>
<td style="text-align: right">1060K</td>
</tr>
<tr>
<td>2590</td>
<td>0</td>
<td>1088</td>
<td style="text-align: right">1138K</td>
</tr>
<tr>
<td>1K</td>
<td>0</td>
<td>585</td>
<td style="text-align: right">783K</td>
</tr>
</tbody>
</table>
<p>在采用kernel的pktgen发包测试中,苏能达到的最大PPS为1200K。2590Bytes文件测试中的PPS已经很接近纯发包测试的极限,所以最终,最充<br />
分利用CPU和带宽的文件大小是2590Bytes。</p>
<h2 id="section-9">测试二</h2>
<p>测试目标:了解 nginx 的大致性能(web服务器模式)</p>
<p>测试工具:ab</p>
<p>测试方法:</p>
<ol>
<li>
<p>S上Nginx做简单配置,使其成为web server;</p>
</li>
<li>
<p>在C1上使用 <测试工具> 对S分别测试空文件、616字节文件(半个包长度)、1232字节文件(即含header字节数为1448,整TCP包长度)、1
233字节文件(超过一个TCP/IP包大小1字节)和2590字节文件;3. 测试中记录S的网络吞吐量和CPU状况;C1的RPS,PPS指标波动情况。</测试工具></p>
</li>
</ol>
<p>测试预期:</p>
<ol>
<li>
<p>吞吐量应超过1Gbps;</p>
</li>
<li>
<p>吞吐量接近5Gbps;</p>
</li>
<li>
<p>适当调整网卡等其他配置,应使RPS超过50万(原有百兆环境下的极限值)。</p>
</li>
</ol>
<p>初步测试:</p>
<p>1、在CentOS6.2系统上,十次测试平均结果发现空文件的RPS仅为458005,响应时间0.88715ms,S带宽121M,未能达到预期。</p>
<p>2、检查发现CentOS6.2的ixgbe驱动版本为3.4.8,与最新版本差距较大,升级ixgbe驱动至最新版3.11.33。</p>
<p>更新后原先的8核client已经无法压满server,更换Client设备,ab并发由50×8提高到50×24,测试不同的InterruptThrottle<br />
Rate条件下数据如下:</p>
<p>1000:</p>
<p><img src="/images/Nginx-testing-10Gibps/c568fc0.png" alt="" /></p>
<p>1500:</p>
<p><img src="/images/Nginx-testing-10Gibps/m15b564f6.png" alt="" /></p>
<p>2500:</p>
<p><img src="/images/Nginx-testing-10Gibps/85b1bbd.png" alt="" /></p>
<p>3500:</p>
<p><img src="/images/Nginx-testing-10Gibps/335545b5.png" alt="" /></p>
<p>4500:</p>
<p><img src="/images/Nginx-testing-10Gibps/m25052f88.png" alt="" /></p>
<p>根据以上数据可知,在ITR1500的情况下,空文件的RPS最高,对比数据如下:</p>
<table>
<thead>
<tr>
<th><strong>ITR</strong></th>
<th><strong>RPS</strong></th>
<th><strong>带宽(MBps)</strong></th>
<th><strong>user%</strong></th>
<th><strong>sys%</strong></th>
<th style="text-align: right"><strong>irq%</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td>1000</td>
<td>688425</td>
<td>186.886</td>
<td>30.112</td>
<td>52.936</td>
<td style="text-align: right">16.951</td>
</tr>
<tr>
<td>1500</td>
<td>691614</td>
<td>187.408</td>
<td>30.125</td>
<td>52.208</td>
<td style="text-align: right">17.667</td>
</tr>
<tr>
<td>2500</td>
<td>682326</td>
<td>183.277</td>
<td>29.191</td>
<td>53.253</td>
<td style="text-align: right">17.556</td>
</tr>
<tr>
<td>3500</td>
<td>650367</td>
<td>175.477</td>
<td>26.803</td>
<td>56.732</td>
<td style="text-align: right">16.382</td>
</tr>
<tr>
<td>4500</td>
<td>630178</td>
<td>169.474</td>
<td>26.417</td>
<td>56.708</td>
<td style="text-align: right">16.833</td>
</tr>
</tbody>
</table>
<p>(注:表格中RPS和带宽数据均为峰值,CPU数据为平稳运行期的中间时刻采样值。下同)</p>
<p>所以在新驱动ITR1500的条件下,重新测试616,1232,1233,2590字节的数据:</p>
<p>616:</p>
<p><img src="/images/Nginx-testing-10Gibps/m2c94d5e6.png" alt="" /></p>
<p>1232:</p>
<p><img src="/images/Nginx-testing-10Gibps/3bd4c0bf.png" alt="" /></p>
<p>1233:</p>
<p><img src="/images/Nginx-testing-10Gibps/m1cb568d3.png" alt="" /></p>
<p>2590</p>
<p><img src="/images/Nginx-testing-10Gibps/m73a60e04.png" alt="" /></p>
<p>测试过程出现剧烈的吞吐量波动,在原有的itr1500的条件下服务极不稳定。</p>
<p>总结ITR1500条件下各文件大小的测试数据对比如下:</p>
<table>
<thead>
<tr>
<th>文件大小(Btyes)</th>
<th>RPS</th>
<th><strong>带宽(MBps)</strong></th>
<th>CPU usr%</th>
<th>CPU sys%</th>
<th style="text-align: right">CPU irq%</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>691614</td>
<td>187.408</td>
<td>30.125</td>
<td>52.208</td>
<td style="text-align: right">17.667</td>
</tr>
<tr>
<td>616</td>
<td>586296</td>
<td>459.435</td>
<td>29.583</td>
<td>54.250</td>
<td style="text-align: right">16.083</td>
</tr>
<tr>
<td>1232</td>
<td>579103</td>
<td>838.146</td>
<td>29.471</td>
<td>50.563</td>
<td style="text-align: right">16.048</td>
</tr>
<tr>
<td>1233</td>
<td>513099</td>
<td>772.826</td>
<td>28.417</td>
<td>48.875</td>
<td style="text-align: right">22.708</td>
</tr>
<tr>
<td>2590</td>
<td>418226</td>
<td>1173.760</td>
<td>23.343</td>
<td>41.934</td>
<td style="text-align: right">18.216</td>
</tr>
</tbody>
</table>
<p>另经测试,ITR上调到15000后,2590字节文件测试的S吞吐量恢复成稳定满带宽运行。而且支持并发到80×24。对比ITR5000/10000/15000<br />
条件下数据:</p>
<p>ITR5000图</p>
<p><img src="/images/Nginx-testing-10Gibps/m7333a2.png" alt="" /></p>
<p>ITR10000图:</p>
<p><img src="/images/Nginx-testing-10Gibps/m7721129e.png" alt="" /></p>
<p>ITR15000图:</p>
<p><img src="/images/Nginx-testing-10Gibps/a58a533.png" alt="" /></p>
<p>2.59KB文件在不同ITR下的测试数据对比如下:</p>
<table>
<thead>
<tr>
<th>ITR</th>
<th>RPS</th>
<th><strong>带宽(MBps)</strong></th>
<th>usr%</th>
<th>sys%</th>
<th>irq%</th>
<th style="text-align: right">错误进程</th>
</tr>
</thead>
<tbody>
<tr>
<td>1500</td>
<td>418226</td>
<td>1173.760</td>
<td>23.343</td>
<td>41.934</td>
<td>18.216</td>
<td style="text-align: right">6</td>
</tr>
<tr>
<td>5000</td>
<td>418216</td>
<td>1172.530</td>
<td>24.250</td>
<td>55.792</td>
<td>19.667</td>
<td style="text-align: right">3</td>
</tr>
<tr>
<td>10000</td>
<td>417989</td>
<td>1171.960</td>
<td>24.625</td>
<td>52.958</td>
<td>22.292</td>
<td style="text-align: right">2</td>
</tr>
<tr>
<td>15000</td>
<td>404330</td>
<td>1132.490</td>
<td>23.802</td>
<td>52.855</td>
<td>23.343</td>
<td style="text-align: right">0</td>
</tr>
</tbody>
</table>
<h2 id="section-10">测试三</h2>
<p>测试目标:了解nginx配置对性能的影响(access_log)</p>
<p>测试工具:ab</p>
<p>测试方法:</p>
<ol>
<li>
<p>S上做简单配置,关闭 access_log 后进行测试;</p>
</li>
<li>
<p>再测试关闭 access_log buffer 的情况(对比测试二的 access_log buffer=1024k 的情况)</p>
</li>
<li>
<p>在C1上使用 <测试工具> 对S分别测试空文件(ITR为1500)</测试工具></p>
</li>
<li>
<p>测试中记录CPU占用率,网络吞吐量,RPS和平均响应时间指标</p>
</li>
</ol>
<p>测试结果:</p>
<p>1、设置<code class="highlighter-rouge">access_log /data/access_log main;</code>的情况:</p>
<p>与关闭日志相比,吞吐量下降,并且无法支持相同数并发(平均有20%的ab进程退出),只能在30×24的并发数情况下完成稳定测试。</p>
<p><img src="/images/Nginx-testing-10Gibps/604d3126.png" alt="" /></p>
<p>2、将日志写入SSD磁盘,验证是否有性能提升。</p>
<p><img src="/images/Nginx-testing-10Gibps/m34c0cd58.png" alt="" /></p>
<p>由上两图对比,可见虽然将日志写入SSD对RPS稍有提升,但离关闭日志的70万差距甚远,更换SSD没有实质的意义。</p>
<p>3、普通磁盘上设置buffer=1024k参数</p>
<p><img src="/images/Nginx-testing-10Gibps/5637f0f7.png" alt="" /></p>
<p>4、设置buffer=64k参数</p>
<p><img src="/images/Nginx-testing-10Gibps/m3d18099f.png" alt="" /></p>
<p>5、设置buffer=4k参数</p>
<p><img src="/images/Nginx-testing-10Gibps/m160f2f0b.png" alt="" /></p>
<p>可见在使用buffer参数后,RPS明显提升到和关闭日志时一个数量级的水准。而buffer的大小,从4k到1024k,也可以提高10%左右。</p>
<p>测试三各项数据对比如下:</p>
<table>
<thead>
<tr>
<th><strong>配置</strong></th>
<th><strong>RPS</strong></th>
<th><strong>带宽(MBps)</strong></th>
<th><strong>Usr%</strong></th>
<th><strong>Sys%</strong></th>
<th><strong>irq%</strong></th>
<th style="text-align: right"><strong>稳定并发</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td>关闭日志</td>
<td>691614</td>
<td>187.408</td>
<td>30.125</td>
<td>52.208</td>
<td>17.667</td>
<td style="text-align: right">50*24</td>
</tr>
<tr>
<td>打开日志</td>
<td>180148</td>
<td>49.074</td>
<td>8.966</td>
<td>21.337</td>
<td>4.483</td>
<td style="text-align: right">30*24</td>
</tr>
<tr>
<td>使用SSD</td>
<td>144347</td>
<td>39.179</td>
<td>10.875</td>
<td>25.708</td>
<td>5.750</td>
<td style="text-align: right">30*24</td>
</tr>
<tr>
<td>buffer4K</td>
<td>588767</td>
<td>159.297</td>
<td>27.613</td>
<td>55.352</td>
<td>16.951</td>
<td style="text-align: right">50*24</td>
</tr>
<tr>
<td>buffer64K</td>
<td>593022</td>
<td>160.172</td>
<td>26.845</td>
<td>57.065</td>
<td>16.048</td>
<td style="text-align: right">50*24</td>
</tr>
<tr>
<td>buffer1M</td>
<td>659290</td>
<td>177.959</td>
<td>32.042</td>
<td>51.875</td>
<td>16.000</td>
<td style="text-align: right">50*24</td>
</tr>
</tbody>
</table>
<h2 id="section-11">测试四</h2>
<p>测试目标:了解nginx配置对性能的影响(tcp_nopush off)</p>
<p>测试工具:ab</p>
<p>测试方法:</p>
<ol>
<li>
<p>S上做简单配置,同时关闭tcp_nopush off;</p>
</li>
<li>
<p>在C1上使用 <测试工具> 对S分别测试1232字节文件和1233字节文件</测试工具></p>
</li>
<li>
<p>测试中记录网络吞吐量,RPS和PPS指标</p>
</li>
</ol>
<p>测试预期:</p>
<p>根据对TCP_NOPUSH的理解,在关闭此参数的情况下,1232字节文件的rps应该和开启状态下有较大差别。</p>
<p>测试结果:</p>
<p>1232字节文件数据:</p>
<p>当ITR为1500时:</p>
<p><img src="/images/Nginx-testing-10Gibps/51d7bf66.png" alt="" /></p>
<p>峰值带宽比测试二中的数据稍差,但是运行不稳定,CPU的波动与rps图相对应,尝试加大并发则有client进程出现connection timeout错误。</p>
<p>加大itr到15000后波动变的比较稳定。但RPS下降到40万,与测试二相差接近30%。数据如下:</p>
<p><img src="/images/Nginx-testing-10Gibps/m21d6c756.png" alt="" /></p>
<p>该项测试相关数据对比如下:</p>
<table>
<thead>
<tr>
<th><strong>配置</strong></th>
<th><strong>RPS</strong></th>
<th><strong>带宽(MBps)</strong></th>
<th><strong>Usr%</strong></th>
<th><strong>Sys%</strong></th>
<th><strong>Irq%</strong></th>
<th style="text-align: right"><strong>退出进程</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td>tcp_nopush on+ITR1500</td>
<td>579103</td>
<td>838.146</td>
<td>29.471</td>
<td>50.563</td>
<td>16.048</td>
<td style="text-align: right">0</td>
</tr>
<tr>
<td>tcp_nopush off+ITR1500</td>
<td>543050</td>
<td>817.746</td>
<td>27.095</td>
<td>51.438</td>
<td>21.467</td>
<td style="text-align: right">1</td>
</tr>
<tr>
<td>tcp_nopush off+ITR15000</td>
<td>393416</td>
<td>593.411</td>
<td>22.458</td>
<td>55.583</td>
<td>21.958</td>
<td style="text-align: right">0</td>
</tr>
</tbody>
</table>
<h2 id="section-12">测试五</h2>
<p>测试目标:了解nginx的大致性能(代理模式)</p>
<p>测试工具:ab</p>
<p>测试方法:</p>
<ol>
<li>
<p>S上nginx做简单配置,使其成为webserver,作为后端;</p>
</li>
<li>
<p>LB上nginx做简单配置,使其成为L7 HTTP 代理,所有请求代理至S。LB后端keepalive在测试中测试两种情况,即打开和关闭</p>
</li>
<li>
<p>在C1/C2上使用 <测试工具> 对LB分别测试空文件、1232字节文件和2590字节文件</测试工具></p>
</li>
<li>
<p>测试中记录S和LB的CPU占用率,网络吞吐量,RPS三个指标</p>
</li>
</ol>
<p>测试预期:</p>
<ol>
<li>
<p>Keepalive关闭时,可能需要对LB的内核参数进行调整才能够完成测试;</p>
</li>
<li>
<p>Keepalive关闭时,仅调整内核参数可能不够,S需绑定多个IP地址,LB需使用S的多个IP地址;</p>
</li>
</ol>
<p>测试结果:</p>
<p>(注:因为C1/C2的主板架构/CPU配置不一。8核的C2会先于24核的C1结束测试,后半段数据不如前半段稳定,不过单C1运行也基本可以逼近LB极限,不影响<br />
数据采集和判断。)</p>
<p>1、空文件:</p>
<p><img src="/images/Nginx-testing-10Gibps/m45af05e8.png" alt="" /></p>
<p>2、1232字节文件</p>
<p><img src="/images/Nginx-testing-10Gibps/4687c6d4.png" alt="" /></p>
<p>3、2590字节文件</p>
<p><img src="/images/Nginx-testing-10Gibps/m449464d4.png" alt="" /></p>
<p>4、10K字节文件</p>
<p><img src="/images/Nginx-testing-10Gibps/m3622cdb6.png" alt="" /></p>
<p>根据测试二的经验,非空文件ITR为15000时表现更稳定,修改后数据如下:</p>
<p>1、1232字节:</p>
<p><img src="/images/Nginx-testing-10Gibps/9098f14.png" alt="" /></p>
<p>2、2590字节:</p>
<p><img src="/images/Nginx-testing-10Gibps/m7d14ebde.png" alt="" /></p>
<p>3、10k字节文件</p>
<p><img src="/images/Nginx-testing-10Gibps/278a77c9.png" alt="" /></p>
<p>4、100k字节文件</p>
<p><img src="/images/Nginx-testing-10Gibps/ffd255.png" alt="" /></p>
<p>关闭proxy_keepalive,使用2台Client测试,并发2400。空文件测试带宽只能压到18MBps,LB和S的CPU<br />
idle都在85%以上。平均响应时间长达40ms。限于环境,压力测试无法继续进行。</p>
<p>本测试proxy_keepalive情况时各项数据对比如下:</p>
<table>
<thead>
<tr>
<th><strong>ITR+文件大小</strong></th>
<th><strong>RPS</strong></th>
<th><strong>带宽(MBps)</strong></th>
<th><strong>Usr%</strong></th>
<th><strong>Sys%</strong></th>
<th style="text-align: right"><strong>Irq%</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td>ITR1500+0Byte</td>
<td>392406</td>
<td>165.185</td>
<td>39.533</td>
<td>26.647</td>
<td style="text-align: right">30.234</td>
</tr>
<tr>
<td>ITR1500+1232Bytes</td>
<td>369582</td>
<td>588.209</td>
<td>41.142</td>
<td>28.595</td>
<td style="text-align: right">28.929</td>
</tr>
<tr>
<td>ITR1500+2590Bytes</td>
<td>296866</td>
<td>898.885</td>
<td>34.250</td>
<td>29.417</td>
<td style="text-align: right">36.167</td>
</tr>
<tr>
<td>ITR1500+10KBytes</td>
<td>118163</td>
<td>1167.280</td>
<td>17.870</td>
<td>18.319</td>
<td style="text-align: right">15.218</td>
</tr>
<tr>
<td>ITR15000+1232Bytes</td>
<td>370590</td>
<td>588.422</td>
<td>41.167</td>
<td>28.375</td>
<td style="text-align: right">29.208</td>
</tr>
<tr>
<td>ITR15000+2590Bytes</td>
<td>297289</td>
<td>897.755</td>
<td>34.890</td>
<td>28.429</td>
<td style="text-align: right">35.682</td>
</tr>
<tr>
<td>ITR15000+10KBytes</td>
<td>118033</td>
<td>1168.160</td>
<td>21.137</td>
<td>21.426</td>
<td style="text-align: right">25.752</td>
</tr>
<tr>
<td>ITR15000+100KBytes</td>
<td>11388</td>
<td>1168.890</td>
<td>4.798</td>
<td>12.542</td>
<td style="text-align: right">10.522</td>
</tr>
</tbody>
</table>
<h2 id="section-13">测试六</h2>
<p>测试目标:nginx LB在多域名情况下的性能</p>
<p>测试工具:ab</p>
<p>测试方法:</p>
<p>1、S上nginx做简单配置,使其成为webserver,作为后端;</p>
<p>2、LB上nginx使用我司实际代理配置(800+域名),使其成为L7 HTTP<br />
代理,所有请求代理至S。LB后端keepalive在测试中测试两种情况,即打开和关闭</p>
<p>3、在C1上使用 <测试工具> 对LB分别测试单域名和随机多域名空文件请求</测试工具></p>
<p>4、测试中记录S和LB的CPU占用率,网络吞吐量,RPS三个指标</p>
<p>测试预期:</p>
<p>多域名代理情况下性能应和直接通过IP访问在一个数量级内。</p>
<p>测试结果:</p>
<p>1、单域名:</p>
<p><img src="/images/Nginx-testing-10Gibps/m76022a04.png" alt="" /></p>
<p>2、随机域名。因为测试在2台服务器上一共开启了32个ab进程,即随机选出32个域名做的测试,结果如下:</p>
<p><img src="/images/Nginx-testing-10Gibps/41f8c534.png" alt="" /></p>
<p>3、关闭keepalive的情况:</p>
<p><img src="/images/Nginx-testing-10Gibps/m6f56712d.png" alt="" /></p>
<p>可见代理模式使用多个domain和upstream配置,对性能没有太大影响。有影响的依然是keepalive是否开启。具体数据对比如下:</p>
<table>
<thead>
<tr>
<th><strong>测试条件</strong></th>
<th><strong>RPS</strong></th>
<th><strong>带宽(MBps)</strong></th>
<th><strong>Usr%</strong></th>
<th><strong>Sys%</strong></th>
<th style="text-align: right"><strong>Irq%</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td>IP访问</td>
<td>392406</td>
<td>165.185</td>
<td>39.533</td>
<td>26.647</td>
<td style="text-align: right">30.234</td>
</tr>
<tr>
<td>单域名</td>
<td>377292</td>
<td>179.832</td>
<td>41.091</td>
<td>26.811</td>
<td style="text-align: right">29.559</td>
</tr>
<tr>
<td>多域名</td>
<td>384513</td>
<td>185.632</td>
<td>43.297</td>
<td>27.186</td>
<td style="text-align: right">28.185</td>
</tr>
<tr>
<td>No keepalive</td>
<td>34879</td>
<td>28.310</td>
<td>5.973</td>
<td>7.341</td>
<td style="text-align: right">9.083</td>
</tr>
</tbody>
</table>
<h2 id="section-14">测试七</h2>
<p>测试目标:其他常见server性能对比</p>
<p>测试工具:ab</p>
<p>测试方法:</p>
<p>Resin4pro加临时lisence,默认配置(关闭日志),测试空文件。</p>
<p>测试结果:</p>
<p><img src="/images/Nginx-testing-10Gibps/m2e7e1790.png" alt="" /></p>
<p>与nginx的webserver模式对比如下:</p>
<table>
<thead>
<tr>
<th><strong>Webserver</strong></th>
<th><strong>RPS</strong></th>
<th><strong>带宽(MBps)</strong></th>
<th><strong>Usr%</strong></th>
<th><strong>Sys%</strong></th>
<th style="text-align: right"><strong>Irq%</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td>resin4</td>
<td>44873</td>
<td>32.353</td>
<td>3.884</td>
<td>4.545</td>
<td style="text-align: right">10.455</td>
</tr>
<tr>
<td>Nginx1.2</td>
<td>691614</td>
<td>187.408</td>
<td>30.125</td>
<td>52.208</td>
<td style="text-align: right">17.667</td>
</tr>
</tbody>
</table>
<h1 id="section-15">术语及缩写说明</h1>
<p>RPS: Requests per Second,每秒请求数</p>
<p>PPS: Packets per second,每秒报文数</p>
<h1 id="section-16">附注</h1>
<p>生成图片的 GNUplot 脚本见<a href="http://chenlinux.com/2012/11/22/gnuplot-to-draw-multi-graph">http://chenlinux.com/2012/11/22/gnuplot-to-draw-multi-graph</a>。</p>
STF 2.0 安装测试
2013-02-22T00:00:00+08:00
perl
http://chenlinux.com/2013/02/22/setup-stf2
<p>STF 更新到 2.0 版本,支持使用 redis 队列做任务分发,比原先的 Q4M 容易上手多了;新增了 cluster 概念,虽然目前看没什么用,不过估计以后肯定要在这方面做文章的。</p>
<p>部署步骤如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="c"># 因为 stf 要求在 Perl5.12 以上运行,CentOS6 还是 5.10 的老版本,所以直接用 Debian 测试了</span>
apt-get install -y memcached redis-server libmysqlclient-dev libdbd-mysql-perl
<span class="c"># 设置 mysql-server 包安装时需要的问答</span>
<span class="nb">echo </span>mysql-server-5.5 mysql-server/root_password <span class="k">select </span>123456 | debconf-set-selections
<span class="nb">echo </span>mysql-server-5.5 mysql-server/root_password_again <span class="k">select </span>123456 | debconf-set-selections
apt-get install mysql-server-5.5
<span class="c"># 系统依赖解决,开始 perl 部分</span>
git clone git://github.com/stf-storage/stf.git
<span class="nb">cd </span>stf
cpanm Redis Data::Dumper::Concise
cpanm --installdeps .
<span class="c"># 创建 mysql 库和用户</span>
mysql -uroot -p -e <span class="s1">'create database stf'</span>
mysql -uroot -p -e <span class="s1">'grant all privileges on stf.* to stf@"%" identified by "654321"'</span>
<span class="c"># 默认监听本机,分布式系统肯定是要放开这个的</span>
sed -i <span class="s1">'s/127.0.0.1/0.0.0.0/'</span> /etc/mysql/my.cnf
service mysql restart
<span class="c"># 导入 sql 建表</span>
mysql -ustf -p stf < misc/stf.sql
<span class="c"># 给 worker 和 dispatcher 设置队列使用 redis</span>
<span class="nb">export </span><span class="nv">STF_QUEUE_TYPE</span><span class="o">=</span>Redis
<span class="nb">export </span><span class="nv">STF_REDIS_HOSTPORT</span><span class="o">=</span>192.168.0.101:6379
<span class="c"># 所有的角色都要有自己独有的 hostid</span>
<span class="nb">export </span><span class="nv">STF_HOST_ID</span><span class="o">=</span>1
<span class="nb">export </span><span class="nv">STF_HOME</span><span class="o">=</span>/root/stf
<span class="c"># 启动 dispatcher,这里目前还只会用 plack,不知道怎么用 nginx/apache</span>
<span class="nb">export </span><span class="nv">USE_PLACK_REPROXY</span><span class="o">=</span>1
<span class="c"># 研究阶段可以打开 debug 看系统是怎么分发怎么平衡怎么确定使用哪个storage的file的过程</span>
<span class="nb">export </span><span class="nv">STF_DEBUG</span><span class="o">=</span>1
plackup -a etc/dispatcher.psgi
<span class="c"># 启动 worker</span>
./bin/stf-worker
<span class="c"># 启动管理界面网站,可以通过 web 添加 cluster 和 storage</span>
plackup -a etc/admin.psgi -p 9000 &
<span class="c"># 一个 cluster 下至少需要有 3 个 storage,这里用三个目录三个端口来模拟</span>
mkdir -p /data<span class="o">{</span>1,2,3<span class="o">}</span>
<span class="nb">export </span><span class="nv">STF_STORAGE_ROOT</span><span class="o">=</span>/data1
plackup -a etc/storage.psgi -p 8888 &
<span class="nb">export </span><span class="nv">STF_STORAGE_ROOT</span><span class="o">=</span>/data2
plackup -a etc/storage.psgi -p 8889 &
<span class="nb">export </span><span class="nv">STF_STORAGE_ROOT</span><span class="o">=</span>/data3
plackup -a etc/storage.psgi -p 8890 &
</code></pre>
</div>
<p>然后上 9000 端口的 web 添加 cluster 和 storage,如下截图:</p>
<p><img src="/images/uploads/stf-admin1.png" alt="cluster" /></p>
<p><img src="/images/uploads/stf-admin.png" alt="storage" /></p>
<p>最后测试一下上传下载,如果上面 psgi 是 DEBUG 运行的,就可以看到详细的过程了。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> lwp-request -m PUT http://192.168.0.101/bucket
^D
lwp-request -m PUT http://192.168.0.101:5000/bucket/test.txt
<span class="nb">test</span>
^D
lwp-request http://192.168.0.101:5000/bucket/test.txt
ls /data1/p/e/g/k/pegkuclninhsyqxftuzpwcuhgughpa.txt
ls /data2/p/e/g/k/pegkuclninhsyqxftuzpwcuhgughpa.txt
</code></pre>
</div>
<p><strong>2013 年 03 月 20 日更新</strong></p>
<p>前面测试记录的,都是纯 perl 的部分。实际运用的时候,有些地方是可以用 nginx 来替代的。</p>
<p>源代码包中,apache-sample.conf 比 nginx-sample.conf 要全面的多。不过其实还是 nginx 配置起来容易,比如给 dispatcher.psgi 加上 nginx 代理,只需要这样就可以了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">server</span> <span class="p">{</span>
<span class="kn">listen</span> <span class="mi">80</span><span class="p">;</span>
<span class="kn">server_name</span> <span class="s">stf</span><span class="p">;</span>
<span class="kn">location</span> <span class="n">/</span> <span class="p">{</span>
<span class="kn">proxy_pass</span> <span class="s">http://192.168.0.101:5000/</span><span class="p">;</span>
<span class="p">}</span>
<span class="kn">location</span> <span class="n">/reproxy</span> <span class="p">{</span>
<span class="kn">internal</span><span class="p">;</span>
<span class="kn">set</span> <span class="nv">$reproxy</span> <span class="nv">$upstream_http_x_reproxy_url</span><span class="p">;</span>
<span class="kn">proxy_pass</span> <span class="nv">$reproxy</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>然后我们就可以直接通过 <code class="highlighter-rouge">http://192.168.0.101/bucket/test.txt</code> 来访问了。</p>
Puppet 自定义 type 和 function
2013-01-31T00:00:00+08:00
devops
puppet
ruby
http://chenlinux.com/2013/01/31/puppet-define-type-and-custom-function
<p>Puppet 除了原有 DSL 以外,还提供了不少接口方便大家开发插件来更简单的完成一些高级功能。</p>
<h1 id="define-type">Define Type</h1>
<p>比如我们要维护一个上千域名组成的 ProxyServer 集群,其域名配置是相近的。那么我们就可以提炼出 template 里会变化的部分作为参数。由此定义出一个 type 如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">define</span> <span class="n">nginx</span><span class="o">::</span><span class="n">vhost4proxy</span><span class="p">(</span>
<span class="vg">$iplist</span> <span class="o">=</span> <span class="p">[],</span>
<span class="vg">$domainlist</span> <span class="o">=</span> <span class="p">[],</span>
<span class="vg">$extconf</span> <span class="o">=</span> <span class="s1">''</span>
<span class="p">)</span> <span class="p">{</span>
<span class="vg">$nginx_proxy_name</span> <span class="o">=</span> <span class="vg">$name</span>
<span class="vg">$nginx_proxy_servers</span> <span class="o">=</span> <span class="vg">$iplist</span>
<span class="vg">$nginx_server_names</span> <span class="o">=</span> <span class="vg">$domainlist</span>
<span class="n">file</span> <span class="p">{</span> <span class="s2">"${nginx_proxy_name}.server.conf"</span><span class="p">:</span>
<span class="k">ensure</span> <span class="o">=></span> <span class="n">file</span><span class="p">,</span>
<span class="nb">require</span> <span class="o">=></span> <span class="no">File</span><span class="p">[</span><span class="s1">'/etc/nginx/conf.d'</span><span class="p">],</span>
<span class="n">path</span> <span class="o">=></span> <span class="s2">"/etc/nginx/conf.d/${nginx_proxy_name}.server.conf"</span><span class="p">,</span>
<span class="n">content</span> <span class="o">=></span> <span class="n">template</span><span class="p">(</span><span class="s1">'nginx/vhost_proxy.conf.erb'</span><span class="p">),</span>
<span class="n">notify</span> <span class="o">=></span> <span class="no">Service</span><span class="p">[</span><span class="s1">'nginx'</span><span class="p">],</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>然后在 template 里使用参数来生成结果:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">upstream</span> <span class="o"><</span><span class="sx">%= nginx_proxy_name %> {
consistent_hash $request_uri;
<% nginx_proxy_servers.each do |ip| -%>
server <%=</span> <span class="n">ip</span> <span class="sx">%>;
<% end %></span>
<span class="p">}</span>
<span class="n">server</span> <span class="p">{</span>
<span class="n">listen</span> <span class="mi">80</span><span class="p">;</span>
<span class="n">server_name</span> <span class="o"><</span><span class="sx">% scope.lookupvar("nginx_server_names").each </span><span class="k">do</span> <span class="o">|</span><span class="nb">name</span><span class="o">|</span> <span class="o">-</span><span class="sx">%> <%= name -%></span><span class="o"><</span><span class="sx">% end </span><span class="o">%></span><span class="p">;</span>
<span class="n">location</span> <span class="o">/</span> <span class="p">{</span>
<span class="n">proxy_pass</span> <span class="n">http</span><span class="ss">:/</span><span class="o">/<</span><span class="sx">%= scope.lookupvar("nginx_proxy_name") %>;
include conf.d/proxy.conf;
}
<% if has_variable?("extconf") %>
<%=</span> <span class="n">scope</span><span class="p">.</span><span class="nf">lookupvar</span><span class="p">(</span><span class="s2">"extconf"</span><span class="p">)</span> <span class="o">%></span>
<span class="o"><</span><span class="sx">% end </span><span class="o">%></span>
<span class="p">}</span>
</code></pre>
</div>
<p>这样我们只需要在 puppet 中这样调用,就可以直接生成对应的配置了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">nginx</span><span class="o">::</span><span class="n">vhost4proxy</span><span class="p">(</span><span class="s1">'server1'</span><span class="p">:</span>
<span class="p">[</span><span class="s1">'1.1.1.1 weight=2'</span><span class="p">,</span> <span class="s1">'2.2.2.2 weight=3'</span><span class="p">],</span>
<span class="p">[</span><span class="s1">'server1.domain'</span><span class="p">,</span> <span class="s1">'server1.alias.domain'</span><span class="p">],</span>
<span class="s1">'access_log /path/to/other_log format'</span>
<span class="p">)</span>
</code></pre>
</div>
<h1 id="custom-function">Custom Function</h1>
<p>不过用上面 define type 还不能完全解决我们提出的问题。因为在 puppet 配置里写几千行 nginx::vhost4proxy 也是一件很可怕的事情!</p>
<p>这时候可以更进一步,把 vhost4proxy 的调用过程隐藏成一个 function,如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">require</span> <span class="s1">'yaml'</span>
<span class="k">module</span> <span class="nn">Puppet::Parser::Functions</span>
<span class="n">newfunction</span><span class="p">(</span><span class="ss">:gen_proxy_confd</span><span class="p">,</span> <span class="ss">:type</span> <span class="o">=></span> <span class="ss">:statement</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">args</span><span class="o">|</span>
<span class="no">Puppet</span><span class="o">::</span><span class="no">Parser</span><span class="o">::</span><span class="no">Functions</span><span class="p">.</span><span class="nf">autoloader</span><span class="p">.</span><span class="nf">loadall</span>
<span class="n">resource_type</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">yaml_dir</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="no">Dir</span><span class="p">.</span><span class="nf">foreach</span><span class="p">(</span><span class="n">yaml_dir</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">yaml_file</span><span class="o">|</span>
<span class="n">file_path</span> <span class="o">=</span> <span class="s2">"</span><span class="si">#{</span><span class="n">yaml_dir</span><span class="si">}</span><span class="s2">/</span><span class="si">#{</span><span class="n">yaml_file</span><span class="si">}</span><span class="s2">"</span>
<span class="k">next</span> <span class="k">unless</span> <span class="n">file_path</span><span class="p">[</span><span class="o">-</span><span class="mi">5</span><span class="p">.</span><span class="nf">.</span><span class="o">-</span><span class="mi">1</span><span class="p">].</span><span class="nf">eql?</span><span class="p">(</span><span class="s1">'.yaml'</span><span class="p">)</span>
<span class="n">res_params</span> <span class="o">=</span> <span class="no">YAML</span><span class="p">.</span><span class="nf">load_file</span><span class="p">(</span><span class="n">file_path</span><span class="p">)</span>
<span class="n">function_create_resources</span><span class="p">([</span><span class="n">resource_type</span><span class="p">,</span> <span class="n">res_params</span><span class="p">])</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p>然后只要把原先传递给 vhost4proxy 的参数写成 yaml 文件放好就行了。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="s">---</span>
<span class="s">server1</span><span class="pi">:</span>
<span class="s">iplist</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">1.1.1.1 weight=2</span>
<span class="pi">-</span> <span class="s">2.2.2.2 weight=3</span>
<span class="s">domainlist</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">server1.domain</span>
<span class="pi">-</span> <span class="s1">'</span><span class="s">*.server1.alias.domain'</span>
<span class="s">extconf</span><span class="pi">:</span> <span class="pi">|-</span>
<span class="no">chunkin on;</span>
<span class="no">error_page 411 = @my_411_error;</span>
<span class="no">location @my_411_error {</span>
<span class="no">chunkin_resume;</span>
<span class="no">}</span>
<span class="no">access_log /path/to/other_log format;</span>
</code></pre>
</div>
<p>大家看起来是不是有点眼熟?没错,这个 yaml 的思路完全是借鉴了 hiera 的写法。但是 hiera 的设计是垂直继承的,不适合这里假设的平面式的情况 —— 当然,如果你觉得把这几千个 yaml 都写在一个大 yaml 文件里也不费劲的话。就不用上我这么折腾了~~</p>
<p>最后在 puppet 配置中只用一行就搞定全部:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">gen_proxy_confd</span><span class="p">(</span><span class="s1">'nginx::vhost4proxy'</span><span class="p">,</span><span class="s2">"${modulepath}/nginx/yaml"</span><span class="p">)</span>
</code></pre>
</div>
<h1 id="section">要点</h1>
<p>type 基本没有什么难度,因为他还是属于 puppet DSL 的运用。可以在其他配置文件内部直接写 define type,不过 puppet-lint 工具会报一个 warnings,所以建议还是单独拆分出来。</p>
<p>function 首先是路径和命名问题。</p>
<ol>
<li>要把写 function 的文件放在 <code class="highlighter-rouge">${modulepath}/yourmodule/lib/puppet/parser/functions/</code> 路径下;</li>
<li>和其他 type、class 一样,文件名必须和 function 一致,puppet 才能 autoload;</li>
<li>格式是固定的,注意有两种:type,statement和rvalue。如果你的 function 目的是返回一个值给 puppet 继续使用,要指定好。默认是 statement;</li>
<li>在自定义 function 里调用其他 function 有两种办法,一种写全路径 <code class="highlighter-rouge">Puppet::Parser::Functions.function('file')</code>;一种是使用 <code class="highlighter-rouge">Puppet::Parser::Functions.autoloader.loadall</code> 加载全部 function,然后用 <code class="highlighter-rouge">function_**</code> 的方式来调用;</li>
<li>示例中最关键的一个是调用了 <code class="highlighter-rouge">function_create_resources</code> 。<code class="highlighter-rouge">create_resources</code> 用来批量创建资源。直接在 puppet 配置文件里使用的时候,接收的是列表参数。但是在 Ruby 里直接使用 <code class="highlighter-rouge">function_create_resources</code> 的话,接收的是一个匿名数组作为唯一参数。</li>
<li>function 和 type 在 puppet 中可以认为是 class 的一种,所以它们也是有自己的作用域的。所以看到传递参数时写的是 “nginx::vhost4proxy”。</li>
</ol>
<h1 id="section-1">参考内容</h1>
<p>关于 Facts 在 function 中的运用,rvalue 的示例等更多内容见官网:<a href="http://docs.puppetlabs.com/guides/custom_functions.html">http://docs.puppetlabs.com/guides/custom_functions.html</a>。</p>
<p>关于 puppet 自带的各种 function 的说明,见官网(很多也没写):<a href="http://docs.puppetlabs.com/references/latest/function.html">http://docs.puppetlabs.com/references/latest/function.html</a>。</p>
<h1 id="section-2">鸣谢</h1>
<p>感谢 <a href="http://weibo.com/liucy1983">@liu.cy</a> 童鞋提醒我变量作用域的问题。function 的调试过程很痛苦。</p>
用 systemtap 调试 kmsg dump
2013-01-11T00:00:00+08:00
monitor
systemtap
http://chenlinux.com/2013/01/11/systemtap-to-debug-kmsg-dump
<p>google 之前推出了一个 netoops 的 patch,可以让 linux kernel 在崩溃的时候通过 udp 协议把信息发送到远端主机上。我之前在 CentOS6.2 的内核上做过测试,详细做法可以参见淘宝内核组 wiki 的<a href="http://kernel.taobao.org/index.php/Documents/Kernel_build">编译使用淘宝内核</a>和 <a href="kernel.taobao.org/index.php/Documents/Kernel_netoops_howto">netoops 使用指南</a>。唯一有区别的地方就是淘宝使用的 RedHat6 的内核在 CentOS6 上有签名问题,需要自己从 CentOS 官网 ftp 下载 src.rpm 来用 —— 当然如果要自己搞定编译那步,少不了就要自己修改 config-genaric 和 kernel.spc 文件了。</p>
<p>昨天同事升级修改到 CentOS6.3 内核( 2.6.32.220 -> 2.6.32.279 )上。结果发现修改冲突代码编译通过后,再使用 soft dump 方式测试,远端主机 nc 收不到结果了。</p>
<p>稍微 grep 一下代码,发现是在 <code class="highlighter-rouge">kernel/printk.c</code> 里定义 <code class="highlighter-rouge">void kmsg_dump()</code> 的。好了,使用 systemtap 来检查这里:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">stap</span> <span class="o">-</span><span class="n">ve</span> <span class="err">'</span><span class="n">probe</span> <span class="n">kernel</span><span class="p">.</span><span class="n">function</span><span class="p">(</span><span class="s">"kmsg_dump"</span><span class="p">){</span><span class="n">printf</span><span class="p">(</span><span class="s">"%s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="err">$$</span><span class="n">vars</span><span class="err">$$</span><span class="p">)}</span><span class="err">'</span>
</code></pre>
</div>
<p>结果发现在 soft dump 的时候有输出,也就是说调用了 <code class="highlighter-rouge">kmsg_dump()</code>。</p>
<p>比较 2.6.32.220 和 2.6.32.279 的代码,发现在 <code class="highlighter-rouge">kmsg_dump()</code> 里,新内核多了一点判断,如果reason 低于 <code class="highlighter-rouge">KERNEL_OOPS</code> 而且没有设置 <code class="highlighter-rouge">always_kmsg_dump</code> 变量,那么直接返回不再 <code class="highlighter-rouge">dumper->dump()</code> 了。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="mi">1546</span> <span class="k">if</span> <span class="p">((</span><span class="n">reason</span> <span class="o">></span> <span class="n">KMSG_DUMP_OOPS</span><span class="p">)</span> <span class="o">&&</span> <span class="o">!</span><span class="n">always_kmsg_dump</span><span class="p">)</span>
<span class="mi">1547</span> <span class="k">return</span><span class="p">;</span>
</code></pre>
</div>
<p>我们验证一下是不是这个原因:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">stap</span> <span class="o">-</span><span class="n">gve</span> <span class="err">'</span><span class="n">probe</span> <span class="n">kernel</span><span class="p">.</span><span class="n">statement</span><span class="p">(</span><span class="s">"*@kernel/printk.c:1548"</span><span class="p">)</span> <span class="p">{</span> <span class="n">printf</span><span class="p">(</span><span class="s">"%s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="err">$$</span><span class="n">parms</span><span class="err">$$</span><span class="p">)</span> <span class="p">}</span><span class="err">'</span>
</code></pre>
</div>
<p>显然测试的时候 reason 是 <code class="highlighter-rouge">KERNEL_SOFT</code>,这个是不好调的,那么我们可以调整这个变量,找了一下没发现这个可以在 sysctl 什么的里面,所以继续用 systemtap 搞定:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">stap</span> <span class="o">-</span><span class="n">gve</span> <span class="err">'</span><span class="n">probe</span> <span class="n">kernel</span><span class="p">.</span><span class="n">statement</span><span class="p">(</span><span class="s">"*@kernel/printk.c:1545"</span><span class="p">)</span> <span class="p">{</span> <span class="err">$</span><span class="n">always_kmsg_dump</span><span class="o">=</span><span class="mi">1</span><span class="p">;</span> <span class="n">printf</span><span class="p">(</span><span class="s">"%d"</span><span class="p">,</span><span class="err">$</span><span class="n">always_kmsg_dump</span><span class="p">);</span> <span class="n">printf</span><span class="p">(</span><span class="s">"%s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="err">$$</span><span class="n">parms</span><span class="err">$$</span><span class="p">)</span> <span class="p">}</span><span class="err">'</span>
</code></pre>
</div>
<p>果然搞定。</p>
升级 Puppet 到 3.0 及其他附件简介
2013-01-10T00:00:00+08:00
devops
puppet
http://chenlinux.com/2013/01/10/update-puppet-to-3.0
<p>今天把 puppet 从2.7 升级到了 3.0。同时放弃了之前通过 ENC 定义所有 top scope variable 的做法,改成只定义一个 role 变量,然后在各个 module 里根据 $role 加载不同的module::role ,把变量都写在 module::role 里。</p>
<p>经历过上次事故后,我对全局变量已经大大的有不安全感,包括 puppet 3.0 新进内核的 hiera (<a href="http://docs.puppetlabs.com/puppet/3/reference/lang_classes.html#using-hierainclude">官网介绍文档</a>中也说是”like a lightweight ENC”)。虽然 module::role 看起来很多是重复内容,还是让人工的操作多经过一些检测才放心。</p>
<p>从 2.7 升级到 3.0 没有太多的不适应。官网上列了很多<a href="http://docs.puppetlabs.com/puppet/3/reference/release_notes.html">不同</a>。不过实际上基本没改动什么。</p>
<ul>
<li>运行命令统一成 <code class="highlighter-rouge">puppet command</code> 的形式,2.7的时候还保留的一堆命令都没有了。</li>
<li><code class="highlighter-rouge">--apply</code> 改成 <code class="highlighter-rouge">--catalog</code> 了。不过这个其实我没用过。</li>
<li><code class="highlighter-rouge">pluginsync</code> 默认开启了。这个是替代 <code class="highlighter-rouge">factsync</code> 的。2.7 的时候默认还是关闭。给 facter 写插件应该是很容易而且很必要的事情。</li>
<li>master 内置 webserver 取消了。也就是说原先各种优化文档里的 <code class="highlighter-rouge">--servertype=mongrel</code> 没用了。但是 3.0 变成了标准 Rack 应用。直接在 <code class="highlighter-rouge">/etc/puppet/rack</code> 下运行 <code class="highlighter-rouge">rackup -s thin -p 18140 -D -P /tmp/puppetmaster.pid</code> 就可以了。</li>
<li>自然对应的 rack 配置文件 <code class="highlighter-rouge">config.ru</code> 改了,看 example 就好。</li>
<li><code class="highlighter-rouge">include</code> 可以传递数组</li>
<li>agent 的 lockfile 把 fork running 和 disabled 区分成两个文件了。不知道能不能消灭掉原先 agent 跑着跑着僵死的情况。</li>
</ul>
<p>以上是官网列举的主要内容。以下还有我__实际测试中发现的问题__:</p>
<ul>
<li>agent 的 puppet.conf 里需要添加一行 <code class="highlighter-rouge">preferred_serialization_format = yaml</code>,否则默认使用 pson 会直接报错。</li>
</ul>
<hr />
<p>今天重温了一下 github 在 puppetconf 上的讲演<a href="https://speakerdeck.com/jnewland/chatops">《chatops》</a>。当然对其中的 hubot 不是重点关注。主要是其中提到的 rodjek 的几个 puppet 相关的项目觉得蛮有用的。</p>
<ul>
<li>puppet-lint</li>
</ul>
<p>地址:<a href="https://github.com/rodjek/puppet-lint.git">https://github.com/rodjek/puppet-lint.git</a></p>
<p>这是一个语法格式检查器,如果 ERROR 会 <code class="highlighter-rouge">exit 1</code>。之前两天我还刚在 CPAN 上发现过一个 <a href="https://metacpan.org/module/Puppet::Tidy">Puppet::Tidy</a> 模块。不过目前为止,这两个都不是很满意:</p>
<ol>
<li>puppet-lint 只能检查格式而不会替你修改格式。</li>
<li>puppet-tidy 可以修改格式但是它对格式的检查太简陋了。</li>
</ol>
<p>当然比 puppet-tidy 稍微好一些的 puppet-lint 也不是很精准,比如他会对所有用双引号定义的变量报 “WARNING: double quoted string containing no variables”;而 puppet-tidy 更奇怪的给我 ip 地址的最后一段再加上了一个单引号变成了下面这个样子:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">$iplist</span> <span class="o">=</span> <span class="p">[</span><span class="s">"192.168.1.'2'"</span><span class="p">,</span><span class="s">"192.168.1.'3'"</span><span class="p">]</span>
</code></pre>
</div>
<p>只能说规范化任重道远。</p>
<ul>
<li>puppet-profiler</li>
</ul>
<p>地址:<a href="https://github.com/rodjek/puppet-profiler.git">https://github.com/rodjek/puppet-profiler.git</a></p>
<p>这是一个 agent 执行的调试器,不过至今为止功能也还很简单:就是执行一次</p>
<div class="highlighter-rouge"><pre class="highlight"><code> puppet agent --test --evaltrace --nocolor
</code></pre>
</div>
<p>排序各个 Resource 的执行耗时,并打印前十名。</p>
<ul>
<li>rspec-puppet</li>
</ul>
<p>地址:<a href="https://github.com/rodjek/rspec-puppet">https://github.com/rodjek/rspec-puppet</a></p>
<p>这是一个 puppet 的 rspec 测试工具扩展。注意他依赖于 <code class="highlighter-rouge">puppetlabs_spec_helper</code> 但是 gem 里却没写。。。</p>
<p>使用方法看 github 上的说明比较详细了,稍后我再单写一篇介绍。</p>
给 puppet 写 Rspec 测试用例
2013-01-10T00:00:00+08:00
devops
puppet
ruby
http://chenlinux.com/2013/01/10/rspec-puppet-intro
<p>上文提到 github 给 puppet 开发的几个附件。其中有扩展 rspec 的 rubygems 模块叫做 rspec-puppet。官网见:<a href="http://rspec-puppet.com">http://rspec-puppet.com</a></p>
<p>照着官网 <a href="http://rspec-puppet.com/tutorial/">Tutorial</a>,很容易能写出来测试用例。我这样ruby入门没看完的水准,从发现这个gem到写完第一个测试用例,也就花了不到半个小时。</p>
<h1 id="section">安装</h1>
<div class="highlighter-rouge"><pre class="highlight"><code> gem install puppetlabs_spec_helper rspec_puppet
</code></pre>
</div>
<h1 id="section-1">创建测试用例环境</h1>
<p>以测试 nginx 模块为例:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">cd</span> /etc/puppet/modules/nginx
rspec-puppet-init
</code></pre>
</div>
<p>这个 init 脚本其实就是执行了一串 <code class="highlighter-rouge">mkdir -p</code> 和 <code class="highlighter-rouge">ln -s</code> 命令,最后生成一个总的 Rakefile 。详情见官网<a href="http://rspec-puppet.com/setup/">Setup</a>。</p>
<h1 id="section-2">编写测试用例</h1>
<p>扩展给 Rspec 增加的方法其实不多,官网 <a href="http://rspec-puppet.com/matchers/">Matchers</a> 页面上有说。主要就是下面几个:</p>
<ul>
<li><code class="highlighter-rouge">include_class()</code></li>
<li><code class="highlighter-rouge">contain_<resource>()</code></li>
<li><code class="highlighter-rouge">run()</code></li>
<li><code class="highlighter-rouge">.with()</code></li>
<li><code class="highlighter-rouge">.without()</code></li>
</ul>
<p>现在来写我们的第一个测试用例 <code class="highlighter-rouge">/etc/puppet/modules/nginx/spec/classes/common_spec.rb</code> 吧:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="c1"># 这个文件被 init 自动生成在 /etc/puppet/modules/nginx/spec/ 下了</span>
<span class="c1"># 其内容就是加入这个目录下所有的文件</span>
<span class="nb">require</span> <span class="s1">'spec_helper'</span>
<span class="c1"># 这里定义你要测试的 puppet module</span>
<span class="n">describe</span> <span class="s1">'nginx'</span> <span class="k">do</span>
<span class="n">it</span> <span class="k">do</span>
<span class="n">should</span> <span class="n">include_class</span><span class="p">(</span><span class="s1">'nginx::sysctl'</span><span class="p">)</span>
<span class="n">should</span> <span class="n">include_class</span><span class="p">(</span><span class="s1">'nginx::install'</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">describe</span> <span class="s1">'nginx::common'</span> <span class="k">do</span>
<span class="c1"># 使用let定义变量</span>
<span class="n">let</span><span class="p">(</span><span class="ss">:node</span><span class="p">)</span> <span class="p">{</span> <span class="s1">'common-nginx-2.domain.com'</span> <span class="p">}</span>
<span class="c1"># 不定义的话,测试中只有从前面:node 生成的 hostname,domain,fqdn 三个</span>
<span class="n">let</span><span class="p">(</span><span class="ss">:facts</span><span class="p">)</span> <span class="p">{</span> <span class="p">{</span>
<span class="ss">:ipaddress_eth0</span> <span class="o">=></span> <span class="s1">'192.168.1.2'</span><span class="p">,</span>
<span class="ss">:processorcount</span> <span class="o">=></span> <span class="s1">'8'</span><span class="p">,</span>
<span class="p">}</span> <span class="p">}</span>
<span class="n">it</span> <span class="k">do</span>
<span class="n">should</span> <span class="n">include_class</span><span class="p">(</span><span class="s1">'nginx::common'</span><span class="p">)</span>
<span class="c1"># 注意这里要写 Resource 的名字,而不是 file 的 path</span>
<span class="c1"># 这个是下面 .with 检查的 :param</span>
<span class="n">should</span> <span class="n">contain_file</span><span class="p">(</span><span class="s1">'proxy.conf'</span><span class="p">).</span><span class="nf">with</span><span class="p">({</span>
<span class="s1">'ensure'</span> <span class="o">=></span> <span class="s1">'file'</span><span class="p">,</span>
<span class="s1">'mode'</span> <span class="o">=></span> <span class="s1">'0644'</span><span class="p">,</span>
<span class="s1">'path'</span> <span class="o">=></span> <span class="s1">'/etc/nginx/conf.d/proxy.conf'</span>
<span class="p">})</span>
<span class="k">end</span>
<span class="n">context</span> <span class="s1">'access_log'</span> <span class="k">do</span>
<span class="n">expect_line</span> <span class="o">=</span> <span class="s1">'access_log /data/nginx/logs/access.log main buffer=16k;'</span>
<span class="n">it</span> <span class="k">do</span>
<span class="c1"># 注意这里是把整个 content 作为 String 对象传递</span>
<span class="n">should</span> <span class="n">contain_file</span><span class="p">(</span><span class="s1">'nginx.conf'</span><span class="p">).</span><span class="nf">with_content</span><span class="p">(</span><span class="sr">/</span><span class="si">#{</span><span class="n">expect_line</span><span class="si">}</span><span class="sr">/</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">context</span> <span class="s1">'upstream'</span> <span class="k">do</span>
<span class="n">expect_line</span> <span class="o">=</span> <span class="s1">'192.168.1.2:80;'</span>
<span class="n">it</span> <span class="k">do</span>
<span class="n">should</span> <span class="n">contain_file</span><span class="p">(</span><span class="s1">'upstream.conf'</span><span class="p">).</span><span class="nf">with_content</span><span class="p">(</span><span class="sr">/</span><span class="si">#{</span><span class="n">expect_line</span><span class="si">}</span><span class="sr">/</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">context</span> <span class="s1">'conf.d'</span> <span class="k">do</span>
<span class="n">it</span> <span class="k">do</span>
<span class="n">dir</span> <span class="o">=</span> <span class="s1">'/etc/puppet/modules/nginx/files/conf.d'</span>
<span class="c1"># eq 是 rspec 本身的方法</span>
<span class="no">Dir</span><span class="p">.</span><span class="nf">entries</span><span class="p">(</span><span class="n">dir</span><span class="p">).</span><span class="nf">length</span><span class="p">.</span><span class="nf">should</span> <span class="n">eq</span><span class="p">(</span><span class="mi">15</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p>然后你就可以运行测试了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">cd</span> /etc/puppet/modules/nginx
rake spec
</code></pre>
</div>
<p>如果测试用例有失败,会在终端看到错误信息。</p>
<p>注意到,rspec 是以 <code class="highlighter-rouge">do ... end</code> 来计算 examples 个数的。在一个 <code class="highlighter-rouge">do ... end</code> 里写多个 should 或者 expect,也算一个 example。</p>
限制单个进程的带宽
2013-01-06T00:00:00+08:00
linux
iptables
cgroups
tc
http://chenlinux.com/2013/01/06/limit-bandwidth-of-one-process
<p>限制带宽简直就是系统管理员的永恒话题之一。当然我这里就不讨论端口限速什么的了,百度一下一大把。但如果要的是限制某个特定进程的带宽,事情就有趣多了。</p>
<h1 id="iptables">iptables</h1>
<p>大多数文档还是提供的传统思路,用 <code class="highlighter-rouge">iptables</code> 的 <code class="highlighter-rouge">owner</code> 模块,给 <code class="highlighter-rouge">--pid-owner</code> 加上 <code class="highlighter-rouge">MARK</code>,然后 <code class="highlighter-rouge">tc</code> 里针对这个 MARK 做限速。用法和限制如 <a href="http://lists.netisland.net/archives/plug/plug-2004-09/msg00454.html">http://lists.netisland.net/archives/plug/plug-2004-09/msg00454.html</a> 说的这样。不过和这个快十年前的文章相比,现在的服务器上,基本已经普及了 SMP ,更进一步的,内核已经在自动发现支持 SMP 的时候,在 iptables 里把 owner 模块的 pid/cmd/sid 三个 match 都去掉了!现在的 owner 里只有 uid/gid 两个。所以这条路,在生产环境上基本行不通。</p>
<p>在 <a href="http://unix.stackexchange.com/questions/34116/how-can-i-limit-the-bandwidth-used-by-a-process">stackexchange</a> 上,大家集思广益、献策献宝,又提出了另外两个工具,那个叫 <code class="highlighter-rouge">pipeviewer</code> 的应用场景比较特定(楼主问题是发生在 sshfs 上),就不多说了。剩下这个 <code class="highlighter-rouge">trickle</code> 真是小众利器。值得一提:</p>
<h1 id="trickle">trickle</h1>
<p>官方主页:<a href="http://monkey.org/~marius/pages/?page=trickle">http://monkey.org/~marius/pages/?page=trickle</a></p>
<p>这是一个在 BSD 上诞生的项目,官网上说只在 i386 的 linux 验证过。不过我在 x86_64 的 linux 替大家尝试了一把,没有问题~</p>
<div class="highlighter-rouge"><pre class="highlight"><code> yum install libevent-devel
wget http://monkey.org/~marius/trickle/trickle-1.06.tar.gz
tar zvxf trickle-1.06.tar.gz
<span class="nb">cd </span>trickle-1.06
./configure
<span class="c"># 生成的 config.h 里重复定义了 in_addr_t 结构体</span>
<span class="c"># 跟 include 的 /usr/include/netinet/in.h 里冲突</span>
<span class="c"># 会报错 "error: two or more data types in declaration specifiers"</span>
sed -i <span class="s1">'s!\(#define in_addr_t\)!//\1!'</span> config.h
make
make install
</code></pre>
</div>
<p>命令使用非常简单:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> trickle -s -d 100 wget http://domain/path/to/file.suffix -O /dev/null
</code></pre>
</div>
<ul>
<li>-s 表示独立运行,因为 trickle 还有一个 trickled 管理端可以用;</li>
<li>-d 表示下载方向;</li>
<li>-u 表示上传方向,两个的单位都是KB/s。</li>
</ul>
<p>这个工具使用了 ELF 的 preloader 机制,在命令执行的时候替换掉标准库中的 socket recv() 和 send() 部分,达到限速的效果。其原理图在<a href="http://monkey.org/~marius/trickle/trickle.pdf">官方PDF</a> 中,如下:</p>
<p><img src="/images/uploads/trickle.png" alt="" /></p>
<p><strong>不过总监大人及时提示我们: 由于该机制的限制,此工具对静态编译的程序无效,对采用 suid 的程序无效!</strong></p>
<h1 id="cgroup">cgroup</h1>
<p>排除上面两个无效,其实 trickle 依然无法覆盖全部应用场景 —— 比如说已经启动的后台进程长期运行,我有 pid ,但是不想中断掉重新起来;或者说这个进程可能我想让他白天跑 10MBps 晚上跑 40MBps 这样动态的。</p>
<p>这个时候就需要动用一些高级工具了,欢迎 <code class="highlighter-rouge">CGROUP</code> 上场。</p>
<p><code class="highlighter-rouge">cgroup</code> 有 <code class="highlighter-rouge">net_cls</code> 控制器。不过和其他控制器不太一样的是它不直接控制网络读写,只是给网络包打上一个标记,然后把专业的事情交给专业的 TC 去做。嗯,思路和原先的 iptable 是很类似的。</p>
<p>参考文档很少,感觉大家使用 cgroup 都集中在 cpu 和 blkio 方面了。目前所见只有 <a href="https://access.redhat.com/knowledge/articles/215353">redhat</a> 这个 pdf:<a href="http://vger.kernel.org/netconf2009_slides/Network%20Control%20Group%20Whitepaper.odt">http://vger.kernel.org/netconf2009_slides/Network%20Control%20Group%20Whitepaper.odt</a> 。实施步骤如下:</p>
<h2 id="tc">启用 tc</h2>
<div class="highlighter-rouge"><pre class="highlight"><code> tc qdisc del dev eth0 root
tc qdisc add dev eth0 root handle 1: htb
tc class add dev eth0 parent 1: classid 1: htb rate 1000mbit ceil 1000mbit
tc class add dev eth0 parent 1: classid 1:3 htb rate 10mbit
tc class add dev eth0 parent 1: classid 1:4 htb rate 10kbit
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 1: cgroup
</code></pre>
</div>
<h2 id="cgroup-1">配置 cgroup</h2>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="c"># 命令行使用</span>
mount -t cgroup net_cls -o net_cls /cgroup/net_cls/
<span class="nb">cd</span> !<span class="err">$</span>
cgcreate -g net_cls:test
<span class="nb">echo</span> <span class="s1">'0x10004'</span> > /cgroup/net_cls/test/net_cls.classid
<span class="c"># 然后可以导出成文件之后通过工具管理</span>
yum install -y libcgroup
cgsnapshot -s > /etc/cgconfig.conf
/etc/init.d/cgconfig restart
</code></pre>
</div>
<h2 id="cgroup-">测试 cgroup 效果</h2>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">time </span>scp bigfile root@192.168.0.26:/tmp/
<span class="nb">time </span>cgexec -g net_cls:test scp bigfile root@192.168.0.26:/tmp/
<span class="nb">echo</span> <span class="nv">$$</span> > /cgroup/net_cls/test/tasks
tc class change dev eth0 parent 1: classid 1:4 htb rate 1mbit
<span class="nb">time </span>scp bigfile root@192.168.0.26:/tmp/
</code></pre>
</div>
<p>可以看到后两次的速度比第一次慢很多。</p>
<p>第三次也被限制住,是因为 cgroup 会自动把子进程的 pid 也加入 tasks 里。</p>
<h1 id="section">总结及其它</h1>
<ul>
<li>trickle 在 download 的时候限制非常管用,在 upload 的时候大概起始速度会比限制值高几倍,然后以 100KB/s 的速度往下减。感觉是 smooth 的问题,不过调整相关参数也没见到区别。</li>
<li>cgroup 给 tc 打标签的办法,看到 tc 限制下的速度波动比较大,猜测 tc 应该是类似 10 秒钟统计一次平均值是否超过限制这样的行为?</li>
</ul>
2012 年个人总结
2012-12-30T00:00:00+08:00
http://chenlinux.com/2012/12/30/report-of-this-year
<p>2012 年还剩下最后 30 个小时。总结一下这一年。</p>
<p>4 月是本年度最重要的一个月,在这个月换工作到了人人,告别了之前八个月乱七八糟的工作状态,再不换,人就要被玩残了。感谢<a href="http://weibo.com/u/1653644220">@懒桃儿吃桃儿</a>的飞速决断。作为自我激励,在换工作的那个周末去搞定了人生大事(好吧,其实压根不是啥激励,应该叫水到渠成)……</p>
<p>9 月是另一个关键点,入职的第一个季度总结,确认自己虽然荒废了八个月,但还没掉队。既然安心自己不至于失业,也就顺带去转职了房奴。</p>
<p>技术博客上还是说点技术点的。</p>
<h1 id="logstash--elasticsearch-">第一、大半年的事情在和 <code class="highlighter-rouge">logstash</code> + <code class="highlighter-rouge">ElasticSearch</code> 系统打交道。</h1>
<p>这个偶然在 oschina 上看到的项目,已经成为我心目中 <code class="highlighter-rouge">splunk</code> 的最佳开源替代品。从 5 月动手测试,到现在为止,写过 2 个相关的 ppt,做过 1 次技术分享,发了 8 篇相关原创博客文章,翻译了 2 篇官网文章,在微博和 QQ 上和大概 10 个左右的朋友交流了搭建、优化的心得。考虑是不是搞个地方存一下这些交流。前几天看 <code class="highlighter-rouge">CloudFoundry</code> 里都有专门的 <code class="highlighter-rouge">ElasticSearch</code> 组件。相信云时代这个穷人版 <code class="highlighter-rouge">splunk</code> 可以走得更远 —— 嗯,不会写 java 用 <code class="highlighter-rouge">hadoop</code> 的运维真的都可以试试。</p>
<h1 id="perl-">第二、Perl 相关。</h1>
<p>本年度关于 <code class="highlighter-rouge">Perl</code> 最多的运用是 <code class="highlighter-rouge">Dancer</code> web开发框架。在 dancer 和 twitter bootstrap 的帮助下做的内部运维工具网站看起来还有那么点意思。使用过程中学会用 IRC 工具和社区进行交流,提供了一个 plugin-upload-progress 的想法,发了两个 patch ,一个给 plugin-flashmessage 同时支持使用 <code class="highlighter-rouge">coderef</code> 的 TT2 和 <code class="highlighter-rouge">object</code> 的 Text::Xslate 模版,一个给 plugin-auth-extensible 的角色认证加上正则匹配的特性。虽然扶凯已经全面转向使用 <code class="highlighter-rouge">Mojolicious</code> 框架了,不过我依然喜欢 dancer 这种广泛使用关键字的方式 —— 少写好多 <code class="highlighter-rouge">$self-></code> 或者 <code class="highlighter-rouge">app-></code> 呢~ 另一个记忆深刻的是某天有人热心的上来说:“我们要多写博客宣传 Dancer 在云计算的运用”,被作者喷还不如给我多写两个插件……</p>
<p>然后是 <code class="highlighter-rouge">Message::Passing</code> 框架,这是 <code class="highlighter-rouge">logstash</code> 项目的 perl port。不过个人纯属好玩和不服气,看完代码当作学习 perl 的 Moo 对象系统了。值得一提是虽然没线上用,倒是找了个周末写了两个模块 <code class="highlighter-rouge">Message::Passing::Filter::Regexp</code> 和 <code class="highlighter-rouge">Message::Passing::Output::PocketIO</code> 上传到 CPAN 了。总算不是光拿不贡献的 perler 了。然后发现 <code class="highlighter-rouge">Test::More</code> 真的蛮不错的,写第二个模块的时候基本就是先写好 test 再写 lib 了,据说这叫 TDD ?</p>
<p>然后是 <code class="highlighter-rouge">Rex</code> 项目,这是一个类似 <code class="highlighter-rouge">func</code> 和 <code class="highlighter-rouge">capistrano</code> 的项目。这类项目和 <code class="highlighter-rouge">puppet</code>、<code class="highlighter-rouge">chef</code>、<code class="highlighter-rouge">cfengine</code> 和 <code class="highlighter-rouge">lcfg</code> 的区别,我觉得是运行目的。cf 系要求的是保持 agent 的配置一致,而 rex 等的目的是给同类目的执行同类任务。这个区别在 2008 年的 sysadmin 里就已经解释过,不过似乎很多人一直在重复问~ 因为 perl 目前还没有像 ruby 的 rakefile 这样流行的任务清单控制(其实有个日本人写的pake,不过在日本人的诸多 modern perl 项目里,我觉得这个 clone 的不咋地),所以我现在都用 Rexfile 来做 task 了,也算一种用法。顺带给只能用 <code class="highlighter-rouge">Net::SSH2</code> 的 Rex 提交了 <code class="highlighter-rouge">Net::OpenSSH</code> 驱动,不过作者很负责任的说要等自己搞出来一套 KerberOS5 认证环境测试我的 pull request 是否能运行后才 merge …… 国内还有哪里用 krb5 认证滴?</p>
<p>最后是 <code class="highlighter-rouge">SmokePing</code> 项目。说实话真没想到这么有名的监控项目代码乱成这个样子。好几次涌现出改写整个项目的疯狂念头,不过看看时间表,想想 rrd 那部分代码看不太懂,放弃了。只好在边边角角上做点修改 —— 顺带再次证明,在 web 开发方面,perl 不是输给了 php/python/ruby ,而是不会 css/js 的 perler 输给了那些会 css/js 的phper/pythoner/rubyer……</p>
<p>附带提一下 <code class="highlighter-rouge">nginx_perl</code> 项目,这个和 <code class="highlighter-rouge">nginx_lua</code> 一样的全流程非阻塞式的东东,最终我还是没找到机会真正用上,白瞎偶去年底很开心的给它写 ppt 推介了。大抵还用 perl 的同学觉得用 <code class="highlighter-rouge">Nginx</code> 内置的阻塞式的 perl 已经够用了?</p>
<h1 id="section">第三、运维、测试</h1>
<p>要感谢人人这个平台,日常运维之余完成了几个测试,<code class="highlighter-rouge">Apache Traffic Server</code> 的因为前设条件比较多,结论不具有普遍性;<code class="highlighter-rouge">Nginx</code> 的万兆网络环境测试还是很有趣的 —— 嗯,虽然最后的脚本依然很难看,拿不出手见人,不过结果还是有力的。<a href="http://weibo.com/u/2266920742">@张纹华</a> ,你的框架要好好搞~</p>
<p><code class="highlighter-rouge">Squid</code> 从我工作以来就在折腾,看样子是要继续折腾下去。感谢 <code class="highlighter-rouge">systemtap</code> 工具,或许明年我在缓慢的学习 C 语言开发的同时,尽量快的搞定那两个问题吧,阿弥陀佛,哈利路亚……</p>
<p><code class="highlighter-rouge">puppet</code> 已经风靡全球,也确实还算好用。我从接触开始就一眼相中了 ENC 接口,不过好像问一圈没谁注意这个。大多数人直接把 hostname 规划和 puppet 连起来了。这部分到底怎样才是 best practice ,还要慢慢看了。</p>
<p><code class="highlighter-rouge">fpm</code> 命令行工具,又是一个 <a href="https://github.com/jordansissel">jordansissel</a> 的 ruby 项目。一般情况下的 package 生成,绝对够用,和 logstash 一样也是我有事没事就推广一下的好东东。</p>
<p><code class="highlighter-rouge">linux</code> 的邮件列表订阅之后迅速的被我过滤掉了,那么多邮件,你们是怎么看过来的,kernel 学习任重道远。</p>
<h1 id="section-1">第四、学习</h1>
<p>上半年,<a href="http://weibo.com/kandeng">@邓侃</a> 博士在北航的云计算公开课基本都去听了。虽然个人工作重点完全不在这方面,不过依然觉得是有所得的。顺带想起某节课后和 <a href="http://weibo.com/ovise">@R_exify</a> 在北航东门外的肯德基里推导怎么设计一个对外透明的 MySQLaaS。大概吹牛就是这样子的……</p>
<p>关于 lua,ruby,javascript,基本都是用到临头抱佛脚,不过对 ruby 和 js 都有深入学习的想法,但是我怨念已久的 C 啊,啥时侯才能学会你。不知道为什么对 python 就是没感觉。话说昨晚在微博上看到有人说看 python 的源码觉得它的 OO 和 lua 很像。顿时我就很郁闷,因为之前我看 lua 的时候觉得 lua 实现的 OO 和 Perl5 很像的好吧。尼玛大家都像来像去的,你们 Pythoner 整天鄙视 Perl 干吗……</p>
<p>关于 CloudFoundry ,关注来关注去,基本停留在看新闻的阶段,一行代码没瞄过,连一个 micro 环境都没搭建过。嗯,不过考虑到毕竟看了不少新闻,还是留一笔。</p>
<p>关于微博,有微博之后,学习成本确实降低了,因为很多问题你可以直接圈人……不过我的同学朋友们肯定对我的微博内容是“眼不见心不烦”了的。哈哈~</p>
<h1 id="section-2">总之,这是无比充实的一年。</h1>
<p>排除掉每个季度末,比如现在,对下季度工作计划的茫然外。</p>
<p>感谢自己依然充满对知识的渴望,让年终的总结显得这么充实和踏实。</p>
perl 模块打包加入外部依赖程序
2012-12-30T00:00:00+08:00
perl
CPAN
http://chenlinux.com/2012/12/30/how-to-install-external-binary-as-perl-modules-dependment
<p>Perl 社区并不是所有的东西都发布在 CPAN 上。甚至专门有一个 <code class="highlighter-rouge">Module::ThirdParty</code> 模块记录这些非 CPAN 的 perl 项目列表。其中最有名的应该就属写博客的 <code class="highlighter-rouge">Movable Type</code> 和做监控的 <code class="highlighter-rouge">SmokePing</code> 了。</p>
<p>但是如果个人图方便又想把 smokeping 打包方便部署使用的时候,就会发现一点小问题:打包成rpm,很多 perl 的依赖模块不一定在系统 repo 里存在;打包成 perl 的模块,smokeping 最常用的几个 probe 比如 fping、curl 什么的,又是非 perl 程序,cpanm 没法解决这个 <code class="highlighter-rouge">requires_external_bin</code> ,最多只能报错退出。</p>
<p>其实这里可以采取一些别的办法,虽然笨一些,但是解决问题。</p>
<p>首先还是让我们创建一个示例模块:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> cpanm Module::Starter Module::Build
module-starter --module Alien::FPing --author<span class="o">=</span><span class="s2">"Jeff Rao"</span> --email<span class="o">=</span><span class="s2">"myname@gmail.com"</span> --mb
</code></pre>
</div>
<p>然后就会在本目录下创建一个 Alien-FPing 目录,自带好了 <code class="highlighter-rouge">Build.PL</code> 等模块文件。这里使用了 <code class="highlighter-rouge">Alien::</code> 的名字空间,是一个潜规则,有些项目依赖 C 源码的库和头文件,就用 perl 包一层来安装,都放在这个空间下,比如 <code class="highlighter-rouge">Alien::V8</code>, <code class="highlighter-rouge">Alien::Gearmand</code>, <code class="highlighter-rouge">Alien::IE7</code> 等等。</p>
<p>现在让我们下载 fping 的源码放到模块里:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> mkdir Alien-FPing/src
wget http://www.fping.org/dist/fping-3.4.tar.gz -O Alien-FPing/src/fping-3.4.tar.gz
</code></pre>
</div>
<p>接下来应该就是编写 <code class="highlighter-rouge">Build.PL</code> 了。不过为了尽量让 <code class="highlighter-rouge">Build.PL</code> 看起来简洁而且一眼看出目的。我们最好把编译操作单独定义一个模块来使用:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">package</span> <span class="nn">Alien::FPing::</span><span class="nv">Build</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">base</span> <span class="sx">qw(Module::Build)</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">File::</span><span class="nv">Spec</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Archive::</span><span class="nv">Tar</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$RootDir</span> <span class="o">=</span> <span class="nn">File::</span><span class="nv">Spec</span><span class="o">-></span><span class="nv">rel2abs</span><span class="p">(</span><span class="s">"."</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$SrcDir</span> <span class="o">=</span> <span class="nn">File::</span><span class="nv">Spec</span><span class="o">-></span><span class="nv">catdir</span><span class="p">(</span><span class="nv">$RootDir</span><span class="p">,</span> <span class="s">"src"</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$FPingVersion</span> <span class="o">=</span> <span class="s">'3.4'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$FPingName</span> <span class="o">=</span> <span class="s">"fping-${FPingVersion}"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$FPingSrc</span> <span class="o">=</span> <span class="s">"${FPingName}.tar.gz"</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">ACTION_build</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$self</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="nb">chdir</span><span class="p">(</span><span class="nv">$SrcDir</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span> <span class="o">!-</span><span class="nv">x</span> <span class="s">"/usr/sbin/fping"</span> <span class="ow">and</span> <span class="o">!-</span><span class="nv">d</span> <span class="nv">$FPingName</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$tar</span> <span class="o">=</span> <span class="nn">Archive::</span><span class="nv">Tar</span><span class="o">-></span><span class="k">new</span><span class="p">();</span>
<span class="nv">$tar</span><span class="o">-></span><span class="nb">read</span><span class="p">(</span><span class="nv">$FPingSrc</span><span class="p">);</span>
<span class="nv">$tar</span><span class="o">-></span><span class="nv">extract</span><span class="p">();</span>
<span class="nb">chdir</span><span class="p">(</span><span class="nv">$FPingName</span><span class="p">);</span>
<span class="nb">system</span><span class="p">(</span><span class="s">'./configure'</span><span class="p">,</span> <span class="s">'--prefix=/usr/'</span><span class="p">,</span> <span class="s">'--enable-ipv6'</span><span class="p">);</span>
<span class="nb">system</span><span class="p">(</span><span class="s">'make'</span><span class="p">);</span>
<span class="nb">system</span><span class="p">(</span><span class="s">'make install'</span><span class="p">);</span>
<span class="p">}</span>
<span class="nv">$self</span><span class="o">-></span><span class="nn">SUPER::</span><span class="nv">ACTION_build</span><span class="p">();</span>
<span class="p">};</span>
<span class="mi">1</span><span class="p">;</span>
</code></pre>
</div>
<p>几乎就是调用 shell 而已,唯一需要讲一下的就是这个 <code class="highlighter-rouge">ACTION_build</code>。这是 <code class="highlighter-rouge">Module::Build</code> 定义好的提供给 <code class="highlighter-rouge">subclass</code> 用的方法,事实上 <code class="highlighter-rouge">./Build help</code> 看得到的所有 action 都有类似的方法可以用。</p>
<p>然后稍微修改一下 Build.PL 如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="mf">5.006</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">warnings</span> <span class="nv">FATAL</span> <span class="o">=></span> <span class="s">'all'</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">lib</span> <span class="s">'inc'</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Alien::FPing::</span><span class="nv">Build</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$builder</span> <span class="o">=</span> <span class="nn">Alien::FPing::</span><span class="nv">Build</span><span class="o">-></span><span class="k">new</span><span class="p">(</span>
<span class="nv">module_name</span> <span class="o">=></span> <span class="s">'Alien::FPing'</span><span class="p">,</span>
<span class="nv">license</span> <span class="o">=></span> <span class="s">'perl'</span><span class="p">,</span>
<span class="nv">dist_author</span> <span class="o">=></span> <span class="sx">q{Jeff Rao <myname@gmail.com>}</span><span class="p">,</span>
<span class="nv">dist_version_from</span> <span class="o">=></span> <span class="s">'lib/Alien/FPing.pm'</span><span class="p">,</span>
<span class="nv">release_status</span> <span class="o">=></span> <span class="s">'stable'</span><span class="p">,</span>
<span class="nv">configure_requires</span> <span class="o">=></span> <span class="p">{</span>
<span class="s">'Module::Build'</span> <span class="o">=></span> <span class="mi">0</span><span class="p">,</span>
<span class="p">},</span>
<span class="nv">build_requires</span> <span class="o">=></span> <span class="p">{</span>
<span class="s">'Test::More'</span> <span class="o">=></span> <span class="mi">0</span><span class="p">,</span>
<span class="p">},</span>
<span class="nv">requires</span> <span class="o">=></span> <span class="p">{</span>
<span class="c1">#'ABC' => 1.6,</span>
<span class="c1">#'Foo::Bar::Module' => 5.0401,</span>
<span class="p">},</span>
<span class="nv">add_to_cleanup</span> <span class="o">=></span> <span class="p">[</span> <span class="s">'Alien-FPing-*'</span> <span class="p">],</span>
<span class="nv">create_makefile_pl</span> <span class="o">=></span> <span class="s">'traditional'</span><span class="p">,</span>
<span class="p">);</span>
<span class="nv">$builder</span><span class="o">-></span><span class="nv">create_build_script</span><span class="p">();</span>
</code></pre>
</div>
<p>把 <code class="highlighter-rouge">Module::Build</code> 替换成 <code class="highlighter-rouge">Alien::FPing::Build</code> 而已,其他都不用动。</p>
<p>然后试一下吧:<br />
<code class="highlighter-rouge">bash
cd Alien-FPing
perl Build.PL
./Build
</code></p>
<p>看到编译输出,并且成功安装有 <code class="highlighter-rouge">/usr/sbin/fping</code> 了吧。现在可以打包了。注意默认生成的 ignore.txt 里,是排除掉了 inc 目录的,需要去除掉,然后修改 <code class="highlighter-rouge">MANIFEST</code> 文件加入 inc 和 src 里的文件,然后再打包出来的 perl 模块就可以直接用了。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> sed -i <span class="s1">'/inc/d'</span> ignore.txt
<span class="nb">echo</span> <span class="s1">'inc/Alien/FPing/Build.pm'</span> >> MANIFEST
<span class="nb">echo</span> <span class="s1">'src/fping-3.4.tar.gz'</span> >> MANIFEST
./Build dist
</code></pre>
</div>
给 Sysadmin Advent 快速搭建本地浏览网站
2012-12-22T00:00:00+08:00
perl
http://chenlinux.com/2012/12/22/simple-local-website-for-sysadmin-advent-clone
<p>一年一度的 advent 集合中,除了 perl 的部分,还有 sysadmin 的也很吸引我等运维的眼球。不过 sysadmin 的一直是发表在blogspot 上,光荣的被 GFW 认证了。虽然说翻墙应该是这年头越来越普及的技能,但是能提供免墙的办法,想来那真真是极好的。</p>
<p>这里提供一个私以为很不错的办法。因为我很开心的发现 sysadvent 有托管在 github 上。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> sudo apt-get install git
git clone git://github.com/jordansissel/sysadvent.git
sudo wget http://xrl.us/cpanm --no-check-certificate -O /sbin/cpanm
sudo chmod +x /sbin/cpanm
cpanm Plack DocLife
</code></pre>
</div>
<p>好了,准备工作完毕。然后在 sysadvent 目录下创建 app.psgi 文件如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nn">Plack::</span><span class="nv">Builder</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Plack::App::</span><span class="nv">Directory</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">DocLife::</span><span class="nv">Markdown</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$html_app</span> <span class="o">=</span> <span class="nn">DocLife::</span><span class="nv">Markdown</span><span class="o">-></span><span class="k">new</span><span class="p">(</span>
<span class="nv">root</span> <span class="o">=></span> <span class="s">'.'</span><span class="p">,</span>
<span class="nv">base_url</span> <span class="o">=></span> <span class="s">'/html/'</span><span class="p">,</span>
<span class="nv">suffix</span> <span class="o">=></span> <span class="s">'.html'</span><span class="p">,</span>
<span class="p">);</span>
<span class="k">my</span> <span class="nv">$md_app</span> <span class="o">=</span> <span class="nn">DocLife::</span><span class="nv">Markdown</span><span class="o">-></span><span class="k">new</span><span class="p">(</span>
<span class="nv">root</span> <span class="o">=></span> <span class="s">'.'</span><span class="p">,</span>
<span class="nv">suffix</span> <span class="o">=></span> <span class="s">'.md'</span><span class="p">,</span>
<span class="nv">base_url</span> <span class="o">=></span> <span class="s">'/md/'</span>
<span class="p">);</span>
<span class="k">my</span> <span class="nv">$dir_app</span> <span class="o">=</span> <span class="nn">Plack::App::</span><span class="nv">Directory</span><span class="o">-></span><span class="k">new</span><span class="p">({</span>
<span class="nv">root</span> <span class="o">=></span> <span class="s">'.'</span><span class="p">,</span>
<span class="p">});</span>
<span class="nv">builder</span> <span class="p">{</span>
<span class="nv">mount</span> <span class="s">'/md'</span> <span class="o">=></span> <span class="nv">$md_app</span><span class="p">;</span>
<span class="nv">mount</span> <span class="s">'/html'</span> <span class="o">=></span> <span class="nv">$html_app</span><span class="p">;</span>
<span class="nv">mount</span> <span class="s">'/'</span> <span class="o">=></span> <span class="nv">builder</span> <span class="p">{</span>
<span class="nv">enable</span> <span class="s">"Plack::Middleware::SimpleContentFilter"</span><span class="p">,</span>
<span class="nv">filter</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">s</span><span class="c1">#(/\d{4}/\d{2}/\S+\.md)#/md\1#;</span>
<span class="nv">s</span><span class="c1">#(/\d{4}/\d{2}/\S+\.html)#/html\1#;</span>
<span class="p">};</span>
<span class="nv">$dir_app</span>
<span class="p">};</span>
<span class="p">};</span>
</code></pre>
</div>
<p><code class="highlighter-rouge">Plack::App::Directory</code> 模块是 <code class="highlighter-rouge">Plack</code> 自带的一个静态目录自动索引发布模块。不过他会把 markdown 当成 “text/plain” 发布,不好看。所以这里引入了另一个 <code class="highlighter-rouge">DocLife</code> 模块。他可以自动把 markdown 和 pod 格式的文档美化转换成 html 格式。本来 <code class="highlighter-rouge">DocLife</code> 本身也提供目录索引功能,不过他的问题是他不考虑 MIME 问题,会把 png 等图片也以 “text/plain” 发布。所以我们用 <code class="highlighter-rouge">Plack::App::URLMap</code> 把两个模块挂在到一起,然后用 <code class="highlighter-rouge">Plack::Middleware::SimpleContentFilter</code> 过滤内容,替换原本的目录链接成针对性的目录。</p>
<p>大功告成!运行命令开始享受世界级运维们的分享吧:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> plackup &
open localhost:5000
</code></pre>
</div>
<table>
<tbody>
<tr>
<td>注:另外有个 <code class="highlighter-rouge">Plack::App::Directory::Markdown</code> 模块,不过他写死了只处理 md,连 html 都被 <code class="highlighter-rouge">next</code>。比较好玩的是这个模块自己把 bootstrap.css</td>
<td>js 给放到 <code class="highlighter-rouge">__DATA__</code> 块里一起分发了,页面倒是更好看一点。</td>
</tr>
</tbody>
</table>
Dancer::Plugin::Adapter 模块介绍
2012-12-22T00:00:00+08:00
dancer
http://chenlinux.com/2012/12/22/intro-dancer-plugin-adapter
<p>Dancer 活跃的社区和强大又方便的插件开发导致出现了太多好玩的插件,有位新同学在刚上手的这两周内就已经往 CPAN 提交了四个插件了。</p>
<p>今天这里介绍一个刚在 IRC 上被推荐的东东,额,这个插件的作者跟上面提到的同学说:大哥,看看偶这个模块吧,就不用你这么辛苦的啥都写新插件了。</p>
<p><a href="https://metacpan.org/module/Dancer::Plugin::Adapter">Dancer::Plugin::Adapter</a> 模块的作用,就是当你的项目需要在多处使用某个模块的时候,不用频繁的到处去new,直接在 config.yml 里一定义,它会自动给你实例化成 <code class="highlighter-rouge">Dancer::Object</code>,然后缓存住,你就可以直接用 service 关键词调用了。</p>
<p>用法示例:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="c1"># in config.yml</span>
<span class="nv">plugins:</span>
<span class="nv">Adapter:</span>
<span class="nv">ua:</span>
<span class="nv">class:</span> <span class="nn">HTTP::</span><span class="nv">Tiny</span>
<span class="nv">options:</span>
<span class="nv">max_redirect:</span> <span class="mi">3</span>
<span class="nv">postmark:</span>
<span class="nv">class:</span> <span class="nn">WWW::</span><span class="nv">Postmark</span>
<span class="nv">options:</span> <span class="nv">POSTMARK_API_TEST</span>
<span class="c1"># in your app</span>
<span class="k">use</span> <span class="nn">Dancer::Plugin::</span><span class="nv">Adapter</span><span class="p">;</span>
<span class="nv">get</span> <span class="s">'/'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="nb">eval</span> <span class="p">{</span>
<span class="nv">service</span><span class="p">(</span><span class="s">"postmark"</span><span class="p">)</span><span class="o">-></span><span class="nb">send</span><span class="p">(</span>
<span class="nv">from</span> <span class="o">=></span> <span class="s">'me@domain.tld'</span><span class="p">,</span>
<span class="nv">to</span> <span class="o">=></span> <span class="s">'you@domain.tld, them@domain.tld'</span><span class="p">,</span>
<span class="nv">subject</span> <span class="o">=></span> <span class="s">'an email message'</span><span class="p">,</span>
<span class="nv">body</span> <span class="o">=></span> <span class="s">"hi guys, what's up?"</span>
<span class="p">);</span>
<span class="p">};</span>
<span class="k">return</span> <span class="vg">$@</span> <span class="p">?</span> <span class="s">"Error: $@"</span> <span class="p">:</span> <span class="s">"Mail sent"</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">get</span> <span class="s">'/proxy/:url'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$res</span> <span class="o">=</span> <span class="nv">service</span><span class="p">(</span><span class="s">'ua'</span><span class="p">)</span><span class="o">-></span><span class="nv">get</span><span class="p">(</span> <span class="nv">params</span><span class="o">-></span><span class="p">{</span><span class="s">'url'</span><span class="p">}</span> <span class="p">);</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$res</span><span class="o">-></span><span class="p">{</span><span class="nv">success</span><span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nv">$res</span><span class="o">-></span><span class="p">{</span><span class="nv">content</span><span class="p">};</span>
<span class="p">}</span>
<span class="k">else</span> <span class="p">{</span>
<span class="nv">template</span> <span class="s">'error'</span> <span class="o">=></span> <span class="p">{</span> <span class="nv">response</span> <span class="o">=></span> <span class="nv">$res</span> <span class="p">};</span>
<span class="p">}</span>
<span class="p">};</span>
</code></pre>
</div>
<p>话说我还是喜欢上代码,不喜欢完整的翻译 POD 啊…………</p>
用 Amcharts 和 ElasticSearch 做日志分析
2012-12-22T00:00:00+08:00
logstash
amcharts
elasticsearch
javascript
http://chenlinux.com/2012/12/22/elasticsearch-amcharts-demo
<p>之前有一篇从 ElasticSearch 官网摘下来的博客<a href="http://chenlinux.com/2012/11/18/data-visualization-with-elasticsearch-and-protovis">《【翻译】用ElasticSearch和Protovis实现数据可视化》</a>。不过一来 Protovis 已经过时,二来 不管是 Protovis 的进化品 D3 还是 Highchart 什么的,我觉得在多图方面都还不如 amcharts 好用。所以在最后依然选择了老牌的 amcharts 完成。</p>
<p>展示品的大概背景还是 webserver 日志,嗯,这个需求应该是最有代表性的了。我们需要对webserver的性能有所了解。之前有一篇文章<a href="http://chenlinux.com/2012/11/22/tatsumaki-demo/">《Tatsumaki框架的小demo一个》</a>,讲的是通过 <code class="highlighter-rouge">terms_stats</code> 获取固定时段内请求时间的平均值。其实这个demo是可以参照官网博客修改成纯js应用的。因为 Tatsumaki 在这里除了处理 HTTP 请求参数,什么都没干。而且这个demo目的是展示 perl 框架的处理,所以amchart方面直接就写死了各种变量。</p>
<p>但是还有一种需求,比如你需要的是针对某个情况超过某个百分比的分时走势统计。这时候必须多次请求 ES 来做运算,再让 js 做,不是说不行,但是多一倍数据在网络中传输,就不如在服务器端封装 API 了 —— 其实是我 js 太烂这种事情,我会告诉你们么。。。</p>
<p>先上两张效果图,其实这个布局我是从 facetgrapher 项目偷来的,但这个项目只适合比较不同 index 之间同时间段的数据,我建议作者修改,作者说”我自己js也是半吊子水平”。。。</p>
<p><img src="/images/uploads/amchart-column.png" alt="分地区错误情况统计" title="分地区错误情况统计" /></p>
<p><img src="/images/uploads/amchart-line.png" alt="实时分运营商错误比例统计" title="实时分运营商错误比例统计" /></p>
<p><strong>2013 年 2 月 21 日更新:利用 bullet 大小来表示 hasErr 的程度</strong></p>
<p>查询的 ES 库情况如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">$ </span>curl <span class="s2">"http://10.4.16.68:9200/demo-photo/log/_mapping?pretty=1"</span>
<span class="o">{</span>
<span class="s2">"log"</span> : <span class="o">{</span>
<span class="s2">"properties"</span> : <span class="o">{</span>
<span class="s2">"brower"</span> : <span class="o">{</span>
<span class="s2">"type"</span> : <span class="s2">"string"</span>
<span class="o">}</span>,
<span class="s2">"date"</span> : <span class="o">{</span>
<span class="s2">"type"</span> : <span class="s2">"date"</span>,
<span class="s2">"format"</span> : <span class="s2">"dateOptionalTime"</span>
<span class="o">}</span>,
<span class="s2">"fromArea"</span> : <span class="o">{</span>
<span class="s2">"type"</span> : <span class="s2">"string"</span>,
<span class="s2">"index"</span> : <span class="s2">"not_analyzed"</span>
<span class="o">}</span>,
<span class="s2">"hasErr"</span> : <span class="o">{</span>
<span class="s2">"type"</span> : <span class="s2">"string"</span>
<span class="o">}</span>,
<span class="s2">"requestUrl"</span> : <span class="o">{</span>
<span class="s2">"type"</span> : <span class="s2">"string"</span>,
<span class="s2">"index"</span> : <span class="s2">"not_analyzed"</span>
<span class="o">}</span>,
<span class="s2">"timeCost"</span> : <span class="o">{</span>
<span class="s2">"type"</span> : <span class="s2">"long"</span>
<span class="o">}</span>,
<span class="s2">"userId"</span> : <span class="o">{</span>
<span class="s2">"type"</span> : <span class="s2">"string"</span>
<span class="o">}</span>,
<span class="s2">"xnforword"</span> : <span class="o">{</span>
<span class="s2">"type"</span> : <span class="s2">"string"</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="nv">$ </span>curl <span class="s2">"http://10.4.16.68:9200/demo-photo/log/_search?pretty=1&size=1"</span> -d <span class="s1">'{"query":{"match_all":{}}}'</span>
<span class="o">{</span>
<span class="s2">"took"</span> : 14,
<span class="s2">"timed_out"</span> : <span class="nb">false</span>,
<span class="s2">"_shards"</span> : <span class="o">{</span>
<span class="s2">"total"</span> : 10,
<span class="s2">"successful"</span> : 10,
<span class="s2">"failed"</span> : 0
<span class="o">}</span>,
<span class="s2">"hits"</span> : <span class="o">{</span>
<span class="s2">"total"</span> : 2330679,
<span class="s2">"max_score"</span> : 1.0,
<span class="s2">"hits"</span> : <span class="o">[</span> <span class="o">{</span>
<span class="s2">"_index"</span> : <span class="s2">"demo-photo"</span>,
<span class="s2">"_type"</span> : <span class="s2">"log"</span>,
<span class="s2">"_id"</span> : <span class="s2">"iSI5xic7Qg2p9Sqk5yp-pQ"</span>,
<span class="s2">"_score"</span> : 1.0, <span class="s2">"_source"</span> : <span class="o">{</span><span class="s2">"hasErr"</span>:<span class="s2">"false"</span>,<span class="s2">"date"</span>:<span class="s2">"2012-12-06T15:04:21,983"</span>,<span class="s2">"userId"</span>:<span class="s2">"123456789"</span>,<span class="s2">"requestUrl"</span>:<span class="s2">"http://photo.demo.domain.com/path/to/your/app/test.jpg"</span>,<span class="s2">"brower"</span>:<span class="s2">"chrome17.0.963.84"</span>,<span class="s2">"timeCost"</span>:750,<span class="s2">"xnforword"</span>:[<span class="s2">"192.168.1.123"</span>,<span class="s2">"10.10.10.10"</span><span class="o">]</span>,<span class="s2">"fromArea"</span>:<span class="s2">"CN-UNI-OTHER"</span><span class="o">}</span>
<span class="o">}</span> <span class="o">]</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre>
</div>
<p>然后后台是我惯用的 Dancer 框架:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">package</span> <span class="nv">AnalysisDemo</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Dancer</span> <span class="s">':syntax'</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Dancer::Plugin::</span><span class="nv">Ajax</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">ElasticSearch</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">POSIX</span> <span class="sx">qw(strftime)</span><span class="p">;</span>
<span class="nb">no</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$elsearch</span> <span class="o">=</span> <span class="nv">ElasticSearch</span><span class="o">-></span><span class="k">new</span><span class="p">(</span> <span class="p">{</span> <span class="nv">%</span><span class="p">{</span> <span class="nv">config</span><span class="o">-></span><span class="p">{</span><span class="nv">plugins</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">ElasticSearch</span><span class="p">}</span> <span class="p">}</span> <span class="p">}</span> <span class="p">);</span>
<span class="k">my</span> <span class="nv">$index_prefix</span> <span class="o">=</span> <span class="s">'demo-'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$type</span> <span class="o">=</span> <span class="s">'log'</span><span class="p">;</span>
<span class="c1"># 这里是对ip库的归类。数据是需要提前导入ES的,这可以是logstash发挥作用</span>
<span class="k">my</span> <span class="nv">$default_provider</span> <span class="o">=</span> <span class="p">{</span>
<span class="nv">yidong</span> <span class="o">=></span> <span class="p">[</span><span class="sx">qw(CN-CRN CN-CMN)</span><span class="p">],</span>
<span class="nv">jiaoyu</span> <span class="o">=></span> <span class="p">[</span><span class="sx">qw(CN-CER CN-CST)</span><span class="p">],</span>
<span class="nv">dianxin</span> <span class="o">=></span> <span class="p">[</span><span class="sx">qw(CN-CHN)</span><span class="p">],</span>
<span class="nv">liantong</span> <span class="o">=></span> <span class="p">[</span><span class="sx">qw(CN-UNI CN-CNC)</span><span class="p">],</span>
<span class="nv">guangdian</span> <span class="o">=></span> <span class="p">[</span><span class="sx">qw(CN-SCN)</span><span class="p">],</span>
<span class="nv">haiwai</span> <span class="o">=></span> <span class="p">[</span><span class="sx">qw(OS)</span><span class="p">],</span>
<span class="p">};</span>
<span class="nv">get</span> <span class="s">'/'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="c1"># 通过 state API 获取 ES 集群现有的所有index列表</span>
<span class="c1"># 因为是一个域名一个index,这样就有了前段页面上的域名下拉选择框</span>
<span class="k">my</span> <span class="nv">$indices</span> <span class="o">=</span> <span class="nv">$elsearch</span><span class="o">-></span><span class="nv">cluster_state</span><span class="o">-></span><span class="p">{</span><span class="nv">routing_table</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">indices</span><span class="p">};</span>
<span class="nv">template</span> <span class="s">'demo/chart'</span><span class="p">,</span>
<span class="p">{</span>
<span class="nv">providers</span> <span class="o">=></span> <span class="p">[</span> <span class="nb">sort</span> <span class="nb">keys</span> <span class="nv">%$default_provider</span> <span class="p">],</span>
<span class="nv">datasources</span> <span class="o">=></span>
<span class="p">[</span> <span class="nb">grep</span> <span class="p">{</span> <span class="sr">/^$index_prefix/</span> <span class="o">&&</span> <span class="sr">s/$index_prefix//</span> <span class="p">}</span> <span class="nb">keys</span> <span class="nv">%$indices</span> <span class="p">],</span>
<span class="nv">inputfrom</span> <span class="o">=></span> <span class="nv">strftime</span><span class="p">(</span><span class="s">"%F\T%T"</span><span class="p">,</span> <span class="nb">localtime</span><span class="p">(</span><span class="nb">time</span><span class="p">()</span><span class="o">-</span><span class="mi">864000</span><span class="p">)),</span>
<span class="nv">inputto</span> <span class="o">=></span> <span class="nv">strftime</span><span class="p">(</span><span class="s">"%F\T%T"</span><span class="p">,</span> <span class="nb">localtime</span><span class="p">()),</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="c1"># 这里把 api 拆成服务商和区域两个,没啥特殊原因,因为是分两回写的,汗</span>
<span class="c1"># 其实可以看到最开始的请求参数类似,最后json的field名字都一样</span>
<span class="nv">ajax</span> <span class="s">'/api/provider'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$param</span> <span class="o">=</span> <span class="nv">from_json</span><span class="p">(</span><span class="nv">request</span><span class="o">-></span><span class="nv">body</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$index</span> <span class="o">=</span> <span class="nv">$index_prefix</span> <span class="o">.</span> <span class="nv">$param</span><span class="o">-></span><span class="p">{</span><span class="s">'datasource'</span><span class="p">};</span>
<span class="k">my</span> <span class="nv">$from</span> <span class="o">=</span> <span class="nv">$param</span><span class="o">-></span><span class="p">{</span><span class="s">'from'</span><span class="p">}</span> <span class="o">||</span> <span class="s">'now-10d'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$to</span> <span class="o">=</span> <span class="nv">$param</span><span class="o">-></span><span class="p">{</span><span class="s">'to'</span><span class="p">}</span> <span class="o">||</span> <span class="s">'now'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$providers</span> <span class="o">=</span> <span class="nv">$param</span><span class="o">-></span><span class="p">{</span><span class="s">'provider'</span><span class="p">};</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$pct</span><span class="p">,</span> <span class="nv">$chartData</span> <span class="p">);</span>
<span class="k">for</span> <span class="k">my</span> <span class="nv">$provider</span> <span class="p">(</span> <span class="nb">sort</span> <span class="nv">@</span><span class="p">{</span><span class="nv">$providers</span><span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$provider_pct</span><span class="p">;</span>
<span class="c1"># 这里是比较麻烦的一点,因为一个区域在ip库里可能标记成多个,比如铁通和移动,现在都是移动</span>
<span class="k">for</span> <span class="k">my</span> <span class="nv">$area</span> <span class="p">(</span> <span class="nv">@</span><span class="p">{</span> <span class="nv">$default_provider</span><span class="o">-></span><span class="p">{</span><span class="nv">$provider</span><span class="p">}</span> <span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$res</span> <span class="o">=</span> <span class="nv">pct_count</span><span class="p">(</span> <span class="nv">$index</span><span class="p">,</span> <span class="nv">$area</span><span class="p">,</span> <span class="nv">$from</span><span class="p">,</span> <span class="nv">$to</span> <span class="p">);</span>
<span class="k">for</span> <span class="k">my</span> <span class="nv">$time</span> <span class="p">(</span> <span class="nb">sort</span> <span class="nb">keys</span> <span class="nv">%</span><span class="p">{</span><span class="nv">$res</span><span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$provider_pct</span><span class="o">-></span><span class="p">{</span><span class="nv">$time</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">count</span><span class="p">}</span> <span class="o">+=</span> <span class="nv">$res</span><span class="o">-></span><span class="p">{</span><span class="nv">$time</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">count</span><span class="p">};</span>
<span class="nv">$provider_pct</span><span class="o">-></span><span class="p">{</span><span class="nv">$time</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">error</span><span class="p">}</span> <span class="o">+=</span> <span class="nv">$res</span><span class="o">-></span><span class="p">{</span><span class="nv">$time</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">error</span><span class="p">};</span>
<span class="nv">$provider_pct</span><span class="o">-></span><span class="p">{</span><span class="nv">$time</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">slow</span><span class="p">}</span> <span class="o">+=</span> <span class="nv">$res</span><span class="o">-></span><span class="p">{</span><span class="nv">$time</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">slow</span><span class="p">};</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1"># 这里因为可能没有错误,所以前面关闭了常用的 warnings 警告</span>
<span class="k">for</span> <span class="k">my</span> <span class="nv">$time</span> <span class="p">(</span> <span class="nb">sort</span> <span class="nb">keys</span> <span class="nv">%</span><span class="p">{</span><span class="nv">$provider_pct</span><span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$right_pct</span> <span class="o">=</span> <span class="mi">100</span><span class="p">;</span>
<span class="nv">$right_pct</span> <span class="o">=</span>
<span class="mi">100</span> <span class="o">-</span>
<span class="nv">$provider_pct</span><span class="o">-></span><span class="p">{</span><span class="nv">$time</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">slow</span><span class="p">}</span> <span class="o">/</span> <span class="nv">$provider_pct</span><span class="o">-></span><span class="p">{</span><span class="nv">$time</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">count</span><span class="p">}</span>
<span class="o">*</span> <span class="mi">100</span><span class="p">;</span>
<span class="nv">$pct</span><span class="o">-></span><span class="p">{</span><span class="nv">$time</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">$provider</span><span class="p">}</span> <span class="o">=</span> <span class="nb">sprintf</span> <span class="s">"%.2f"</span><span class="p">,</span> <span class="nv">$right_pct</span><span class="p">;</span>
<span class="nv">$pct</span><span class="o">-></span><span class="p">{</span><span class="nv">$time</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="s">"${provider}Err"</span><span class="p">}</span> <span class="o">=</span> <span class="nb">sprintf</span> <span class="s">"%.2f"</span><span class="p">,</span>
<span class="nv">$provider_pct</span><span class="o">-></span><span class="p">{</span><span class="nv">$time</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">error</span><span class="p">}</span> <span class="o">/</span> <span class="nv">$provider_pct</span><span class="o">-></span><span class="p">{</span><span class="nv">$time</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">count</span><span class="p">}</span>
<span class="o">*</span> <span class="mi">100</span><span class="p">;</span>
<span class="nv">$pct</span><span class="o">-></span><span class="p">{</span><span class="nv">$time</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="s">"${provider}Size"</span><span class="p">}</span> <span class="o">=</span> <span class="nb">sprintf</span> <span class="s">"%.0f"</span><span class="p">,</span>
<span class="nv">$pct</span><span class="o">-></span><span class="p">{</span><span class="nv">$time</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="s">"${provider}Err"</span><span class="p">};</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="k">for</span> <span class="k">my</span> <span class="nv">$time</span> <span class="p">(</span> <span class="nb">sort</span> <span class="nb">keys</span> <span class="nv">%$pct</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$data</span><span class="o">-></span><span class="p">{</span><span class="nv">date</span><span class="p">}</span> <span class="o">=</span> <span class="nv">$time</span><span class="p">;</span>
<span class="k">for</span> <span class="k">my</span> <span class="nv">$provider</span> <span class="p">(</span> <span class="nv">@$providers</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$data</span><span class="o">-></span><span class="p">{</span><span class="nv">$provider</span><span class="p">}</span> <span class="o">=</span> <span class="nv">$pct</span><span class="o">-></span><span class="p">{</span><span class="nv">$time</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">$provider</span><span class="p">}</span> <span class="o">||</span> <span class="mi">100</span><span class="p">;</span>
<span class="nv">$data</span><span class="o">-></span><span class="p">{</span><span class="s">"${provider}Err"</span><span class="p">}</span> <span class="o">=</span> <span class="nv">$pct</span><span class="o">-></span><span class="p">{</span><span class="nv">$time</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="s">"${provider}Err"</span><span class="p">}</span> <span class="o">||</span> <span class="mi">0</span><span class="p">;</span>
<span class="c1"># 百分比太低,所以翻 5 倍来作为 bullet 的大小</span>
<span class="nv">$data</span><span class="o">-></span><span class="p">{</span><span class="s">"${provider}Size"</span><span class="p">}</span> <span class="o">=</span>
<span class="nv">$pct</span><span class="o">-></span><span class="p">{</span><span class="nv">$time</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="s">"${provider}Size"</span><span class="p">}</span> <span class="o">*</span> <span class="mi">5</span> <span class="o">||</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">};</span>
<span class="nb">push</span> <span class="nv">@$chartData</span><span class="p">,</span> <span class="nv">$data</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">my</span> <span class="nv">$res</span> <span class="o">=</span> <span class="p">{</span>
<span class="nv">type</span> <span class="o">=></span> <span class="s">"line"</span><span class="p">,</span>
<span class="nv">categoryField</span> <span class="o">=></span> <span class="s">"date"</span><span class="p">,</span>
<span class="nv">graphList</span> <span class="o">=></span> <span class="nv">$providers</span><span class="p">,</span>
<span class="nv">chartData</span> <span class="o">=></span> <span class="nv">$chartData</span><span class="p">,</span>
<span class="p">};</span>
<span class="k">return</span> <span class="nv">to_json</span><span class="p">(</span><span class="nv">$res</span><span class="p">);</span>
<span class="p">};</span>
<span class="nv">ajax</span> <span class="s">'/api/area'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$param</span> <span class="o">=</span> <span class="nv">from_json</span><span class="p">(</span><span class="nv">request</span><span class="o">-></span><span class="nv">body</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$index</span> <span class="o">=</span> <span class="nv">$index_prefix</span> <span class="o">.</span> <span class="nv">$param</span><span class="o">-></span><span class="p">{</span><span class="s">'datasource'</span><span class="p">};</span>
<span class="k">my</span> <span class="nv">$limit</span> <span class="o">=</span> <span class="nv">$param</span><span class="o">-></span><span class="p">{</span><span class="s">'limit'</span><span class="p">}</span> <span class="o">||</span> <span class="mi">50</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$from</span> <span class="o">=</span> <span class="nv">$param</span><span class="o">-></span><span class="p">{</span><span class="s">'from'</span><span class="p">}</span> <span class="o">||</span> <span class="s">'now-10d'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$to</span> <span class="o">=</span> <span class="nv">$param</span><span class="o">-></span><span class="p">{</span><span class="s">'to'</span><span class="p">}</span> <span class="o">||</span> <span class="s">'now'</span><span class="p">;</span>
<span class="c1"># 这是后来写的,尽可能把 sub 拆分了,所以 ajax 这里就很简略</span>
<span class="c1"># 当然因为不考虑多运营商的问题,本身也容易一些</span>
<span class="k">my</span> <span class="nv">$res</span> <span class="o">=</span> <span class="nv">pct_terms</span><span class="p">(</span> <span class="nv">$index</span><span class="p">,</span> <span class="nv">$limit</span><span class="p">,</span> <span class="nv">$from</span><span class="p">,</span> <span class="nv">$to</span> <span class="p">);</span>
<span class="k">return</span> <span class="nv">to_json</span><span class="p">(</span><span class="nv">$res</span><span class="p">);</span>
<span class="p">};</span>
<span class="k">sub </span><span class="nf">pct_terms</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$index</span><span class="p">,</span> <span class="nv">$limit</span><span class="p">,</span> <span class="nv">$from</span><span class="p">,</span> <span class="nv">$to</span> <span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$area_all_count</span> <span class="o">=</span> <span class="nv">area_terms</span><span class="p">(</span> <span class="nv">$index</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="nv">$limit</span><span class="p">,</span> <span class="nv">$from</span><span class="p">,</span> <span class="nv">$to</span> <span class="p">);</span>
<span class="k">my</span> <span class="nv">$area_err_count</span> <span class="o">=</span> <span class="nv">area_terms</span><span class="p">(</span> <span class="nv">$index</span><span class="p">,</span> <span class="mi">2000</span><span class="p">,</span> <span class="nv">$limit</span><span class="p">,</span> <span class="nv">$from</span><span class="p">,</span> <span class="nv">$to</span> <span class="p">);</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$error</span><span class="p">,</span> <span class="nv">$chartData</span> <span class="p">);</span>
<span class="k">for</span> <span class="p">(</span> <span class="nv">@</span><span class="p">{</span><span class="nv">$area_err_count</span><span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$error</span><span class="o">-></span><span class="p">{</span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="nv">term</span><span class="p">}</span> <span class="p">}</span> <span class="o">=</span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="nv">count</span><span class="p">};</span>
<span class="p">}</span>
<span class="k">for</span> <span class="p">(</span> <span class="nv">@</span><span class="p">{</span><span class="nv">$area_all_count</span><span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="nb">push</span> <span class="nv">@$chartData</span><span class="p">,</span> <span class="p">{</span>
<span class="nv">area</span> <span class="o">=></span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="nv">term</span><span class="p">},</span>
<span class="nv">error</span> <span class="o">=></span> <span class="nv">$error</span><span class="o">-></span><span class="p">{</span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="nv">term</span><span class="p">}</span> <span class="p">}</span> <span class="o">||</span> <span class="mi">0</span><span class="p">,</span>
<span class="nv">right</span> <span class="o">=></span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="nv">count</span><span class="p">}</span> <span class="o">-</span> <span class="nv">$error</span><span class="o">-></span><span class="p">{</span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="nv">term</span><span class="p">}</span> <span class="p">},</span>
<span class="p">};</span>
<span class="p">}</span>
<span class="k">my</span> <span class="nv">$res</span> <span class="o">=</span> <span class="p">{</span>
<span class="nv">type</span> <span class="o">=></span> <span class="s">"column"</span><span class="p">,</span>
<span class="nv">categoryField</span> <span class="o">=></span> <span class="s">"area"</span><span class="p">,</span>
<span class="nv">graphList</span> <span class="o">=></span> <span class="p">[</span><span class="sx">qw(right error)</span><span class="p">],</span>
<span class="nv">chartData</span> <span class="o">=></span> <span class="nv">$chartData</span><span class="p">,</span>
<span class="p">};</span>
<span class="k">return</span> <span class="nv">$res</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">pct_count</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$index</span><span class="p">,</span> <span class="nv">$area</span><span class="p">,</span> <span class="nv">$from</span><span class="p">,</span> <span class="nv">$to</span> <span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$level</span> <span class="o">=</span> <span class="nv">$area</span> <span class="ow">eq</span> <span class="s">'OS'</span> <span class="p">?</span> <span class="mi">3000</span> <span class="p">:</span> <span class="mi">2000</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$all_count</span> <span class="o">=</span> <span class="nv">histo_count</span><span class="p">(</span> <span class="nv">$index</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="nv">$area</span><span class="p">,</span> <span class="nv">$from</span><span class="p">,</span> <span class="nv">$to</span> <span class="p">);</span>
<span class="k">my</span> <span class="nv">$slow_count</span> <span class="o">=</span> <span class="nv">histo_count</span><span class="p">(</span> <span class="nv">$index</span><span class="p">,</span> <span class="nv">$level</span><span class="p">,</span> <span class="nv">$area</span><span class="p">,</span> <span class="nv">$from</span><span class="p">,</span> <span class="nv">$to</span> <span class="p">);</span>
<span class="k">my</span> <span class="nv">$err_count</span> <span class="o">=</span> <span class="nv">histo_count</span><span class="p">(</span> <span class="nv">$index</span><span class="p">,</span> <span class="s">'hasErr'</span><span class="p">,</span> <span class="nv">$area</span><span class="p">,</span> <span class="nv">$from</span><span class="p">,</span> <span class="nv">$to</span> <span class="p">);</span>
<span class="k">my</span> <span class="nv">$res</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span> <span class="nv">@</span><span class="p">{</span><span class="nv">$slow_count</span><span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$res</span><span class="o">-></span><span class="p">{</span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="nb">time</span><span class="p">}</span> <span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">slow</span><span class="p">}</span> <span class="o">=</span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="nv">count</span><span class="p">};</span>
<span class="p">}</span>
<span class="k">for</span> <span class="p">(</span> <span class="nv">@</span><span class="p">{</span><span class="nv">$err_count</span><span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$res</span><span class="o">-></span><span class="p">{</span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="nb">time</span><span class="p">}</span> <span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">error</span><span class="p">}</span> <span class="o">=</span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="nv">count</span><span class="p">};</span>
<span class="p">}</span>
<span class="k">for</span> <span class="p">(</span> <span class="nv">@</span><span class="p">{</span><span class="nv">$all_count</span><span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$res</span><span class="o">-></span><span class="p">{</span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="nb">time</span><span class="p">}</span> <span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">count</span><span class="p">}</span> <span class="o">=</span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="nv">count</span><span class="p">};</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nv">$res</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1"># 下面开始的两个才是真正发 ES 请求的地方</span>
<span class="k">sub </span><span class="nf">area_terms</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$index</span><span class="p">,</span> <span class="nv">$level</span><span class="p">,</span> <span class="nv">$limit</span><span class="p">,</span> <span class="nv">$from</span><span class="p">,</span> <span class="nv">$to</span> <span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$data</span> <span class="o">=</span> <span class="nv">$elsearch</span><span class="o">-></span><span class="nv">search</span><span class="p">(</span>
<span class="nb">index</span> <span class="o">=></span> <span class="nv">$index</span><span class="p">,</span>
<span class="nv">type</span> <span class="o">=></span> <span class="nv">$type</span><span class="p">,</span>
<span class="nv">size</span> <span class="o">=></span> <span class="mi">0</span><span class="p">,</span>
<span class="nv">facets</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">area</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">facet_filter</span> <span class="o">=></span> <span class="p">{</span>
<span class="ow">and</span> <span class="o">=></span> <span class="p">[</span>
<span class="p">{</span>
<span class="nv">range</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">date</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">from</span> <span class="o">=></span> <span class="nv">$from</span><span class="p">,</span>
<span class="nv">to</span> <span class="o">=></span> <span class="nv">$to</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nv">numeric_range</span> <span class="o">=></span>
<span class="p">{</span> <span class="nv">timeCost</span> <span class="o">=></span> <span class="p">{</span> <span class="nv">gte</span> <span class="o">=></span> <span class="nv">$level</span><span class="p">,</span> <span class="p">},</span> <span class="p">},</span>
<span class="p">},</span>
<span class="p">],</span>
<span class="p">},</span>
<span class="c1"># 使用最简单的 terms facets API,因为只用计数就好了</span>
<span class="nv">terms</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">field</span> <span class="o">=></span> <span class="s">"fromArea"</span><span class="p">,</span>
<span class="nv">size</span> <span class="o">=></span> <span class="nv">$limit</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">);</span>
<span class="k">return</span> <span class="nv">$data</span><span class="o">-></span><span class="p">{</span><span class="nv">facets</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">area</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">terms</span><span class="p">};</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">histo_count</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$index</span><span class="p">,</span> <span class="nv">$level</span><span class="p">,</span> <span class="nv">$area</span><span class="p">,</span> <span class="nv">$from</span><span class="p">,</span> <span class="nv">$to</span> <span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="c1"># 根据 level 参数判断使用 hasErr 还是 timeCost 列数据</span>
<span class="k">my</span> <span class="nv">$level_ref</span> <span class="o">=</span>
<span class="nv">$level</span> <span class="ow">eq</span> <span class="s">'hasErr'</span>
<span class="p">?</span> <span class="p">{</span> <span class="nv">term</span> <span class="o">=></span> <span class="p">{</span> <span class="nv">hasErr</span> <span class="o">=></span> <span class="s">'true'</span> <span class="p">}</span> <span class="p">}</span>
<span class="p">:</span> <span class="p">{</span> <span class="nv">numeric_range</span> <span class="o">=></span> <span class="p">{</span> <span class="nv">timeCost</span> <span class="o">=></span> <span class="p">{</span> <span class="ow">gt</span> <span class="o">=></span> <span class="nv">$level</span> <span class="p">}</span> <span class="p">}</span> <span class="p">};</span>
<span class="k">my</span> <span class="nv">$facets</span> <span class="o">=</span> <span class="p">{</span>
<span class="nv">pct</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">facet_filter</span> <span class="o">=></span> <span class="p">{</span>
<span class="c1"># 这里条件比较多,所以要用 bool API,不能用 and 了</span>
<span class="nv">bool</span> <span class="o">=></span> <span class="p">{</span>
<span class="c1"># must 可以提供多个条件作为 AND 数组</span>
<span class="c1"># 此外还有 must_not 作为 AND NOT 数组</span>
<span class="c1"># should 作为 OR 数组</span>
<span class="nv">must</span> <span class="o">=></span> <span class="p">[</span>
<span class="p">{</span>
<span class="nv">range</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">date</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">from</span> <span class="o">=></span> <span class="nv">$from</span><span class="p">,</span>
<span class="nv">to</span> <span class="o">=></span> <span class="nv">$to</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">{</span> <span class="nv">prefix</span> <span class="o">=></span> <span class="p">{</span> <span class="nv">fromArea</span> <span class="o">=></span> <span class="nv">$area</span> <span class="p">}</span> <span class="p">},</span>
<span class="nv">$level_ref</span><span class="p">,</span>
<span class="p">],</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="c1"># 这里是需要针对专门的时间列做汇总,所以用 date_histogram 了,具体说明之前有博客</span>
<span class="nv">date_histogram</span> <span class="o">=></span> <span class="p">{</span>
<span class="nv">field</span> <span class="o">=></span> <span class="s">"date"</span><span class="p">,</span>
<span class="nv">interval</span> <span class="o">=></span> <span class="s">"1h"</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="k">my</span> <span class="nv">$data</span> <span class="o">=</span> <span class="nv">$elsearch</span><span class="o">-></span><span class="nv">search</span><span class="p">(</span>
<span class="nb">index</span> <span class="o">=></span> <span class="nv">$index</span><span class="p">,</span>
<span class="nv">type</span> <span class="o">=></span> <span class="nv">$type</span><span class="p">,</span>
<span class="nv">facets</span> <span class="o">=></span> <span class="nv">$facets</span><span class="p">,</span>
<span class="nv">size</span> <span class="o">=></span> <span class="mi">0</span><span class="p">,</span>
<span class="p">);</span>
<span class="k">return</span> <span class="nv">$data</span><span class="o">-></span><span class="p">{</span><span class="nv">facets</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">pct</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="nv">entries</span><span class="p">};</span>
<span class="p">}</span>
</code></pre>
</div>
<p>其实把里面请求的hash拆开来一个个定义,然后根据情况组合,但是不方便察看作为 demo 的整体情况。</p>
<p>然后看template里怎么写。这里虽然有两个效果图,但是只有一个template哟:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nt"><link</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"[% $request.uri_base %]/amcharts/style.css"</span> <span class="na">type=</span><span class="s">"text/css"</span><span class="nt">></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"[% $request.uri_base %]/amcharts/amcharts.js"</span> <span class="na">type=</span><span class="s">"text/javascript"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">type=</span><span class="s">"text/javascript"</span><span class="nt">></span>
<span class="kd">var</span> <span class="nx">chart</span><span class="p">;</span>
<span class="kd">function</span> <span class="nx">createAmChart</span><span class="p">(</span><span class="nx">data</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// 清空原有图形</span>
<span class="nx">$</span><span class="p">(</span><span class="s2">"#chartdiv"</span><span class="p">).</span><span class="nx">empty</span><span class="p">();</span>
<span class="c1">// 如果是时间轴线图,需要把date字符转成Date对象</span>
<span class="k">if</span> <span class="p">(</span> <span class="nx">data</span><span class="p">.</span><span class="nx">categoryField</span> <span class="o">==</span> <span class="s2">"date"</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">for</span> <span class="p">(</span> <span class="kd">var</span> <span class="nx">j</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">j</span> <span class="o"><</span> <span class="nx">data</span><span class="p">.</span><span class="nx">chartData</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span> <span class="nx">j</span><span class="o">++</span> <span class="p">)</span> <span class="p">{</span>
<span class="nx">data</span><span class="p">.</span><span class="nx">chartData</span><span class="p">[</span><span class="nx">j</span><span class="p">].</span><span class="nx">date</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Date</span><span class="p">(</span><span class="nb">Number</span><span class="p">(</span><span class="nx">data</span><span class="p">.</span><span class="nx">chartData</span><span class="p">[</span><span class="nx">j</span><span class="p">].</span><span class="nx">date</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nx">chart</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">AmCharts</span><span class="p">.</span><span class="nx">AmSerialChart</span><span class="p">();</span>
<span class="c1">// 拖动条等图片的路径</span>
<span class="nx">chart</span><span class="p">.</span><span class="nx">pathToImages</span> <span class="o">=</span> <span class="s2">"/amcharts/images/"</span><span class="p">;</span>
<span class="nx">chart</span><span class="p">.</span><span class="nx">dataProvider</span> <span class="o">=</span> <span class="nx">data</span><span class="p">.</span><span class="nx">chartData</span><span class="p">;</span>
<span class="nx">chart</span><span class="p">.</span><span class="nx">categoryField</span> <span class="o">=</span> <span class="nx">data</span><span class="p">.</span><span class="nx">categoryField</span><span class="p">;</span>
<span class="c1">// 如果是柱状图,可以显示 3D 效果</span>
<span class="k">if</span> <span class="p">(</span> <span class="nx">data</span><span class="p">.</span><span class="nx">type</span> <span class="o">==</span> <span class="s1">'column'</span> <span class="p">)</span> <span class="p">{</span>
<span class="c1">// chart.rotate = true;</span>
<span class="nx">chart</span><span class="p">.</span><span class="nx">depth3D</span> <span class="o">=</span> <span class="mi">20</span><span class="p">;</span>
<span class="nx">chart</span><span class="p">.</span><span class="nx">angle</span> <span class="o">=</span> <span class="mi">30</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">var</span> <span class="nx">categoryAxis</span> <span class="o">=</span> <span class="nx">chart</span><span class="p">.</span><span class="nx">categoryAxis</span><span class="p">;</span>
<span class="nx">categoryAxis</span><span class="p">.</span><span class="nx">fillAlpha</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="nx">categoryAxis</span><span class="p">.</span><span class="nx">fillColor</span> <span class="o">=</span> <span class="s2">"#FAFAFA"</span><span class="p">;</span>
<span class="nx">categoryAxis</span><span class="p">.</span><span class="nx">axisAlpha</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="nx">categoryAxis</span><span class="p">.</span><span class="nx">gridPosition</span> <span class="o">=</span> <span class="s2">"start"</span><span class="p">;</span>
<span class="c1">// 时间轴需要解析Date对象</span>
<span class="k">if</span> <span class="p">(</span> <span class="nx">data</span><span class="p">.</span><span class="nx">categoryField</span> <span class="o">==</span> <span class="s2">"date"</span> <span class="p">)</span> <span class="p">{</span>
<span class="nx">categoryAxis</span><span class="p">.</span><span class="nx">parseDates</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
<span class="nx">categoryAxis</span><span class="p">.</span><span class="nx">minPeriod</span> <span class="o">=</span> <span class="s2">"hh"</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">var</span> <span class="nx">valueAxis</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">AmCharts</span><span class="p">.</span><span class="nx">ValueAxis</span><span class="p">();</span>
<span class="nx">valueAxis</span><span class="p">.</span><span class="nx">dashLength</span> <span class="o">=</span> <span class="mi">5</span><span class="p">;</span>
<span class="nx">valueAxis</span><span class="p">.</span><span class="nx">axisAlpha</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="c1">// 指定柱状图为叠加模式,这里有多种模式可以看文档</span>
<span class="k">if</span> <span class="p">(</span> <span class="nx">data</span><span class="p">.</span><span class="nx">type</span> <span class="o">==</span> <span class="s1">'column'</span> <span class="p">)</span> <span class="p">{</span>
<span class="nx">valueAxis</span><span class="p">.</span><span class="nx">stackType</span> <span class="o">=</span> <span class="s2">"regular"</span><span class="p">;</span>
<span class="p">}</span>
<span class="nx">chart</span><span class="p">.</span><span class="nx">addValueAxis</span><span class="p">(</span><span class="nx">valueAxis</span><span class="p">);</span>
<span class="c1">// 这里有个有趣的事情,如果不把graph当数组直接循环,效果也没问题</span>
<span class="c1">// 我只能猜测是 addGraph 后数据其实已经缓存到 chart 了</span>
<span class="kd">var</span> <span class="nx">graph</span> <span class="o">=</span> <span class="p">[];</span>
<span class="kd">var</span> <span class="nx">colors</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'#FF6600'</span><span class="p">,</span> <span class="s1">'#FCD202'</span><span class="p">,</span> <span class="s1">'#B0DE09'</span><span class="p">,</span> <span class="s1">'#0D8ECF'</span><span class="p">,</span> <span class="s1">'#2A0CD0'</span><span class="p">,</span> <span class="s1">'#CD0D74'</span><span class="p">,</span> <span class="s1">'#CC0000'</span><span class="p">,</span> <span class="s1">'#00CC00'</span><span class="p">,</span> <span class="s1">'#0000CC'</span><span class="p">,</span> <span class="s1">'#DDDDDD'</span><span class="p">,</span> <span class="s1">'#999999'</span><span class="p">,</span> <span class="s1">'#333333'</span><span class="p">,</span> <span class="s1">'#990000'</span><span class="p">];</span>
<span class="k">for</span> <span class="p">(</span> <span class="kd">var</span> <span class="nx">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="o"><</span> <span class="nx">data</span><span class="p">.</span><span class="nx">graphList</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span> <span class="p">)</span> <span class="p">{</span>
<span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">AmCharts</span><span class="p">.</span><span class="nx">AmGraph</span><span class="p">();</span>
<span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">title</span> <span class="o">=</span> <span class="nx">data</span><span class="p">.</span><span class="nx">graphList</span><span class="p">[</span><span class="nx">i</span><span class="p">];</span>
<span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">valueField</span> <span class="o">=</span> <span class="nx">data</span><span class="p">.</span><span class="nx">graphList</span><span class="p">[</span><span class="nx">i</span><span class="p">];</span>
<span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">type</span> <span class="o">=</span> <span class="nx">data</span><span class="p">.</span><span class="nx">type</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span> <span class="nx">data</span><span class="p">.</span><span class="nx">type</span> <span class="o">==</span> <span class="s1">'column'</span> <span class="p">)</span> <span class="p">{</span>
<span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">lineAlpha</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">fillAlphas</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">valueField</span> <span class="o">=</span> <span class="nx">data</span><span class="p">.</span><span class="nx">graphList</span><span class="p">[</span><span class="nx">i</span><span class="p">];</span>
<span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">descriptionField</span> <span class="o">=</span> <span class="nx">data</span><span class="p">.</span><span class="nx">graphList</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="o">+</span> <span class="s2">"Err"</span><span class="p">;</span>
<span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">bulletSizeField</span> <span class="o">=</span> <span class="nx">data</span><span class="p">.</span><span class="nx">graphList</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="o">+</span> <span class="s2">"Size"</span><span class="p">;</span>
<span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">bullet</span> <span class="o">=</span> <span class="s2">"round"</span><span class="p">;</span>
<span class="c1">// 设定为空心圆圈</span>
<span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">bulletColor</span> <span class="o">=</span> <span class="s2">"#ffffff"</span><span class="p">;</span>
<span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">bulletBorderAlpha</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="c1">// amchart 本来有默认颜色,不过前面因为修改了圆内的颜色,所以其他颜色无法继承默认设定了</span>
<span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">bulletBorderColor</span> <span class="o">=</span> <span class="nx">colors</span><span class="p">[</span><span class="nx">i</span><span class="p">];</span>
<span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">lineColor</span> <span class="o">=</span> <span class="nx">colors</span><span class="p">[</span><span class="nx">i</span><span class="p">];</span>
<span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">lineAlpha</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">lineThickness</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">balloonText</span> <span class="o">=</span> <span class="s2">"[[value]]% / hasErr:[[description]]%"</span><span class="p">;</span>
<span class="p">}</span>
<span class="nx">chart</span><span class="p">.</span><span class="nx">addGraph</span><span class="p">(</span><span class="nx">graph</span><span class="p">[</span><span class="nx">i</span><span class="p">]);</span>
<span class="p">}</span>
<span class="c1">// 加图例,这样可以在图上随时勾选察看具体某个数据,也方便某数据异常的时候影响察看其他</span>
<span class="kd">var</span> <span class="nx">legend</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">AmCharts</span><span class="p">.</span><span class="nx">AmLegend</span><span class="p">();</span>
<span class="nx">legend</span><span class="p">.</span><span class="nx">position</span> <span class="o">=</span> <span class="s2">"right"</span><span class="p">;</span>
<span class="nx">legend</span><span class="p">.</span><span class="nx">horizontalGap</span> <span class="o">=</span> <span class="mi">10</span><span class="p">;</span>
<span class="nx">legend</span><span class="p">.</span><span class="nx">switchType</span> <span class="o">=</span> <span class="s2">"v"</span><span class="p">;</span>
<span class="nx">chart</span><span class="p">.</span><span class="nx">addLegend</span><span class="p">(</span><span class="nx">legend</span><span class="p">);</span>
<span class="c1">// 加拖拉轴,这样可以拖动察看细节,这个功能很赞</span>
<span class="kd">var</span> <span class="nx">scrollbar</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">AmCharts</span><span class="p">.</span><span class="nx">ChartScrollbar</span><span class="p">();</span>
<span class="nx">scrollbar</span><span class="p">.</span><span class="nx">graph</span> <span class="o">=</span> <span class="nx">graph</span><span class="p">[</span><span class="mi">0</span><span class="p">];</span>
<span class="nx">scrollbar</span><span class="p">.</span><span class="nx">graphType</span> <span class="o">=</span> <span class="s2">"line"</span><span class="p">;</span>
<span class="nx">scrollbar</span><span class="p">.</span><span class="nx">height</span> <span class="o">=</span> <span class="mi">30</span><span class="p">;</span>
<span class="nx">chart</span><span class="p">.</span><span class="nx">addChartScrollbar</span><span class="p">(</span><span class="nx">scrollbar</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">cursor</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">AmCharts</span><span class="p">.</span><span class="nx">ChartCursor</span><span class="p">();</span>
<span class="nx">chart</span><span class="p">.</span><span class="nx">addChartCursor</span><span class="p">(</span><span class="nx">cursor</span><span class="p">);</span>
<span class="nx">chart</span><span class="p">.</span><span class="nx">write</span><span class="p">(</span><span class="s2">"chartdiv"</span><span class="p">);</span>
<span class="p">};</span>
<span class="kd">function</span> <span class="nx">drawChart</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">provider</span> <span class="o">=</span> <span class="p">[];</span>
<span class="nx">$</span><span class="p">(</span><span class="s2">"#provider :selected"</span><span class="p">).</span><span class="nx">each</span><span class="p">(</span><span class="kd">function</span><span class="p">(){</span>
<span class="nx">provider</span><span class="p">.</span><span class="nx">push</span><span class="p">(</span> <span class="nx">$</span><span class="p">(</span><span class="k">this</span><span class="p">).</span><span class="nx">val</span><span class="p">()</span> <span class="p">);</span>
<span class="p">});</span>
<span class="kd">var</span> <span class="nx">datasource</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="s2">"#datasource :selected"</span><span class="p">).</span><span class="nx">val</span><span class="p">();</span>
<span class="kd">var</span> <span class="nx">apitype</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="s2">":radio:checked"</span><span class="p">).</span><span class="nx">val</span><span class="p">();</span>
<span class="kd">var</span> <span class="nx">from</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="s2">"#from"</span><span class="p">).</span><span class="nx">val</span><span class="p">();</span>
<span class="kd">var</span> <span class="nx">to</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="s2">"#to"</span><span class="p">).</span><span class="nx">val</span><span class="p">();</span>
<span class="nx">$</span><span class="p">.</span><span class="nx">ajax</span><span class="p">({</span>
<span class="na">processData</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="na">url</span><span class="p">:</span> <span class="s2">"[% $request.uri_base %]/demo/api/"</span> <span class="o">+</span> <span class="nx">apitype</span><span class="p">,</span>
<span class="na">data</span><span class="p">:</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">stringify</span><span class="p">({</span><span class="s2">"provider"</span><span class="p">:</span><span class="nx">provider</span><span class="p">,</span> <span class="s2">"datasource"</span><span class="p">:</span><span class="nx">datasource</span><span class="p">,</span> <span class="s2">"from"</span><span class="p">:</span><span class="nx">from</span><span class="p">,</span> <span class="s2">"to"</span><span class="p">:</span><span class="nx">to</span><span class="p">}),</span>
<span class="na">type</span><span class="p">:</span> <span class="s2">"POST"</span><span class="p">,</span>
<span class="na">dataType</span><span class="p">:</span> <span class="s2">"json"</span><span class="p">,</span>
<span class="na">success</span> <span class="p">:</span> <span class="nx">createAmChart</span>
<span class="p">});</span>
<span class="p">};</span>
<span class="kd">function</span> <span class="nx">showselect</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">(</span><span class="s2">"#providers"</span><span class="p">).</span><span class="nx">show</span><span class="p">();</span>
<span class="p">};</span>
<span class="kd">function</span> <span class="nx">hideselect</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">(</span><span class="s2">"#providers"</span><span class="p">).</span><span class="nx">hide</span><span class="p">();</span>
<span class="p">};</span>
<span class="nt"></script></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"well"</span><span class="nt">></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"span8"</span><span class="nt">></span>
<span class="nt"><input</span> <span class="na">type=</span><span class="s">"text"</span> <span class="na">class=</span><span class="s">"input-medium"</span> <span class="na">id=</span><span class="s">"from"</span> <span class="na">name=</span><span class="s">"from"</span> <span class="na">value=</span><span class="s">"[% $inputfrom %]"</span><span class="nt">></span>
<span class="nt"><input</span> <span class="na">type=</span><span class="s">"text"</span> <span class="na">class=</span><span class="s">"input-medium"</span> <span class="na">id=</span><span class="s">"to"</span> <span class="na">name=</span><span class="s">"to"</span> <span class="na">value=</span><span class="s">"[% $inputto %]"</span><span class="nt">></span>
<span class="nt"><select</span> <span class="na">class=</span><span class="s">"input-medium"</span> <span class="na">id=</span><span class="s">"datasource"</span><span class="nt">></span>
%% for $datasources -> $datasource {
<span class="nt"><option</span> <span class="na">value=</span><span class="s">"[% $datasource %]"</span><span class="nt">></span>[% $datasource %]<span class="nt"></option></span>
%% }
<span class="nt"></select></span>
<span class="nt"></div></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"span2"</span><span class="nt">></span>
<span class="nt"><label</span> <span class="na">class=</span><span class="s">"radio"</span><span class="nt">></span>
<span class="nt"><input</span> <span class="na">type=</span><span class="s">"radio"</span> <span class="na">name=</span><span class="s">"querytype"</span> <span class="na">value=</span><span class="s">"provider"</span> <span class="na">onclick=</span><span class="s">"showselect()"</span><span class="nt">></span>服务商趋势
<span class="nt"></label></span>
<span class="nt"><label</span> <span class="na">class=</span><span class="s">"radio"</span><span class="nt">></span>
<span class="nt"><input</span> <span class="na">type=</span><span class="s">"radio"</span> <span class="na">name=</span><span class="s">"querytype"</span> <span class="na">value=</span><span class="s">"area"</span> <span class="na">checked</span> <span class="na">onclick=</span><span class="s">"hideselect()"</span><span class="nt">></span>分地区统计
<span class="nt"></label></span>
<span class="nt"></div></span>
<span class="nt"><button</span> <span class="na">type=</span><span class="s">"submit"</span> <span class="na">class=</span><span class="s">"btn btn-primary"</span> <span class="na">onclick=</span><span class="s">"drawChart()"</span><span class="nt">></span>查询<span class="nt"></button></span>
<span class="nt"><div</span> <span class="na">id =</span><span class="s">"providers"</span> <span class="na">class=</span><span class="s">"controls hide"</span><span class="nt">></span>
<span class="nt"><select</span> <span class="na">class=</span><span class="s">"input-medium"</span> <span class="na">id=</span><span class="s">"provider"</span> <span class="na">multiple=</span><span class="s">"mulitiple"</span><span class="nt">></span>
%% for $providers -> $provider {
<span class="nt"><option</span> <span class="na">value=</span><span class="s">"[% $provider %]"</span> <span class="na">selected</span><span class="nt">></span>[% $provider %]<span class="nt"></option></span>
%% }
<span class="nt"></select></span>
<span class="nt"></div></span>
<span class="nt"></div></span><span class="c"><!--/well--></span>
<span class="nt"><div</span> <span class="na">id=</span><span class="s">"chartdiv"</span> <span class="na">style=</span><span class="s">"width: 100%; height: 400px;"</span><span class="nt">></span>
<span class="nt"></div></span>
</code></pre>
</div>
学习 Dancer::Plugin::Auth::Extensible 模块
2012-12-21T00:00:00+08:00
dancer
http://chenlinux.com/2012/12/21/dancer-plugin-auth-extensible
<p>首先介绍一下 <code class="highlighter-rouge">Dancer::Plugin::Auth::Extensible</code> 模块。这是一个认证验证的框架,之前 Dancer 里这方面的框架是 RBAC ,不过 RBAC 是实现的 auth 对象,然后提供 <code class="highlighter-rouge">->asa</code>,<code class="highlighter-rouge">->can</code>,<code class="highlighter-rouge">->roles</code> 等方法。在使用的时候,需要自己在每个 route 里写 if 或者 switch 代码,显得比较繁琐。而 Extensible 模块提供了另一个(或者说是两个)思路。同时借此深入了解 <code class="highlighter-rouge">Dancer::Plugin</code> 和 <code class="highlighter-rouge">Dancer::Hook</code> 的用法,外加熟悉 perl 的一些不常见的对象使用。收获良多,不可不记。</p>
<p>上面之所以说算是两个思路。是因为在这个模块出来的短短十天内,其 0.001 和 0.010 版本已经完全从实现到使用方法都变了样子。下面先说 0.001 版。</p>
<p>这个原始版本的使用方法大概是这样的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">get</span> <span class="s">'/secret'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">:RequireRole(God) {</span> <span class="nv">DestroyWorld</span><span class="p">();</span> <span class="p">};</span>
<span class="nv">get</span> <span class="s">'/users'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">:RequireLogin {</span>
<span class="k">my</span> <span class="nv">$user</span> <span class="o">=</span> <span class="nv">logged_in_user</span><span class="p">;</span>
<span class="k">return</span> <span class="s">"Hi there, $user->{username}"</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<p>哇,我是第一次见到在 <code class="highlighter-rouge">sub</code> 后面还可以写这样的东西(好吧,暴露了本人的菜鸟本质)!赶紧打开模块的源代码,然后找到了相关的几行:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nv">attributes</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Scalar::</span><span class="nv">Util</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Exporter</span> <span class="s">'import'</span><span class="p">;</span>
<span class="k">our</span> <span class="nv">@EXPORT</span><span class="o">=</span><span class="sx">qw(MODIFY_CODE_ATTRIBUTES FETCH_CODE_ATTRIBUTES)</span><span class="p">;</span>
<span class="nv">hook</span> <span class="nv">before</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$route_handler</span> <span class="o">=</span> <span class="nb">shift</span> <span class="o">||</span> <span class="k">return</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$requires_login</span> <span class="o">=</span> <span class="nv">get_attribs_by_type</span><span class="p">(</span>
<span class="s">'RequireLogin'</span><span class="p">,</span> <span class="nv">$route_handler</span><span class="o">-></span><span class="nv">code</span>
<span class="p">);</span>
<span class="k">my</span> <span class="nv">$roles_required</span> <span class="o">=</span> <span class="nv">get_attribs_by_type</span><span class="p">(</span>
<span class="s">'RequireRole'</span><span class="p">,</span> <span class="nv">$route_handler</span><span class="o">-></span><span class="nv">code</span>
<span class="p">);</span>
<span class="o">...</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">my</span> <span class="nv">%attrs</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">MODIFY_CODE_ATTRIBUTES</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$package</span><span class="p">,</span> <span class="nv">$subref</span><span class="p">,</span> <span class="nv">@attrs</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="nv">$attrs</span><span class="p">{</span> <span class="nv">refaddr</span> <span class="nv">$subref</span> <span class="p">}</span> <span class="o">=</span> <span class="o">\</span><span class="nv">@attrs</span><span class="p">;</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">FETCH_CODE_ATTRIBUTES</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$package</span><span class="p">,</span> <span class="nv">$subref</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$attrs</span> <span class="o">=</span> <span class="nv">$attrs</span><span class="p">{</span> <span class="nv">refaddr</span> <span class="nv">$subref</span> <span class="p">};</span>
<span class="k">return</span> <span class="nv">$attrs</span> <span class="p">?</span> <span class="nv">@$attrs</span> <span class="p">:</span> <span class="p">();</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">get_attribs_by_type</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$type</span><span class="p">,</span> <span class="nv">$coderef</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">return</span> <span class="k">unless</span> <span class="nv">$coderef</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@desired_attribs</span> <span class="o">=</span> <span class="nb">grep</span> <span class="p">{</span>
<span class="sr">/^$type(?:\([^)]*\))?$/</span>
<span class="p">}</span> <span class="nn">attributes::</span><span class="nv">get</span><span class="p">(</span><span class="nv">$coderef</span><span class="p">);</span>
<span class="k">return</span> <span class="k">if</span> <span class="o">!</span><span class="nv">@desired_attribs</span><span class="p">;</span>
<span class="k">return</span> <span class="p">[</span>
<span class="nb">map</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$f</span> <span class="o">=</span> <span class="nv">$_</span><span class="p">;</span>
<span class="nv">$f</span> <span class="o">=~</span> <span class="sr">s/^$type\(\s*([^)]*)\s*\)$/$1/</span><span class="p">;</span>
<span class="nb">split</span><span class="p">(</span><span class="sr">/\s+/</span><span class="p">,</span> <span class="nv">$f</span><span class="p">);</span>
<span class="p">}</span> <span class="nv">@desired_attribs</span>
<span class="p">];</span>
<span class="p">}</span>
</code></pre>
</div>
<p>代码中的 <code class="highlighter-rouge">$route_handler->code</code> 就是应用中写的 <code class="highlighter-rouge">sub {}</code>。<strong>整个代码中,最关键的部分是这句 <code class="highlighter-rouge">attributes::get($coderef)</code> !</strong></p>
<p>首先有个小问题,因为 Dancer 里,get 是关键词,所以这里写了全路径。<code class="highlighter-rouge">attributes::get</code> 的介绍见 <a href="https://metacpan.org/module/attributes#Available-Subroutines">POD</a>,大意是会使用 <code class="highlighter-rouge">FETCH_type_ATTRIBUTES</code> 方法获取列表。因为这里 attribute 是 sub 的,所以 type 就是 CODE ,也就是用前面定义的 <code class="highlighter-rouge">FETCH_CODE_ATTRIBUTES</code>。<code class="highlighter-rouge">FETCH_type_ATTRIBUTES</code> 方法的说明见 <a href="https://metacpan.org/module/attributes#Package-specific-Attribute-Handling">POD</a>。</p>
<p>在<a href="https://metacpan.org/module/perlsub#Subroutine-Attributes">https://metacpan.org/module/perlsub#Subroutine-Attributes</a>中,建议我们看另一个更好用的模块来理解自定义属性的问题,这个模块是<a href="https://metacpan.org/module/Attribute::Handlers">Attribute::Handlers</a>。</p>
<p>然后是 0.010 版:</p>
<p>新版本的使用方法如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">get</span> <span class="s">'/secret'</span> <span class="o">=></span> <span class="nv">require_any_role</span> <span class="p">[</span><span class="sx">qw(God Admin)</span><span class="p">]</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="nv">DestroyWorld</span><span class="p">();</span> <span class="p">};</span>
<span class="nv">get</span> <span class="s">'/users'</span> <span class="o">=></span> <span class="nv">require_login</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$user</span> <span class="o">=</span> <span class="nv">logged_in_user</span><span class="p">;</span>
<span class="k">return</span> <span class="s">"Hi there, $user->{username}"</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<p>这种添加新关键词的写法更加的 dancer。所以能从实现中学到更有普适性的 <code class="highlighter-rouge">Dancer::Plugin</code> 开发方法。摘要代码如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nn">Dancer::</span><span class="nv">Plugin</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Dancer</span> <span class="sx">qw(:syntax)</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">require_any_role</span> <span class="p">{</span>
<span class="k">return</span> <span class="nv">_build_wrapper</span><span class="p">(</span><span class="nv">@_</span><span class="p">,</span> <span class="s">'any'</span><span class="p">);</span>
<span class="p">}</span>
<span class="nv">register</span> <span class="nv">require_any_role</span> <span class="o">=></span> <span class="o">\&</span><span class="nv">require_any_role</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">_build_wrapper</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$require_role</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$coderef</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$mode</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@role_list</span> <span class="o">=</span> <span class="nb">ref</span> <span class="nv">$require_role</span> <span class="ow">eq</span> <span class="s">'ARRAY'</span>
<span class="p">?</span> <span class="nv">@$require_role</span>
<span class="p">:</span> <span class="nv">$require_role</span><span class="p">;</span>
<span class="k">return</span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$user</span> <span class="o">=</span> <span class="nv">logged_in_user</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nv">$user</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">execute_hook</span><span class="p">(</span><span class="s">'login_required'</span><span class="p">,</span> <span class="nv">$coderef</span><span class="p">);</span>
<span class="k">return</span> <span class="nv">redirect</span> <span class="nv">$loginpage</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">my</span> <span class="nv">$role_match</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$mode</span> <span class="ow">eq</span> <span class="s">'single'</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$role_match</span><span class="o">++</span> <span class="k">if</span> <span class="nv">user_has_role</span><span class="p">(</span><span class="nv">$require_role</span><span class="p">);</span>
<span class="p">}</span> <span class="k">elsif</span> <span class="p">(</span><span class="nv">$mode</span> <span class="ow">eq</span> <span class="s">'any'</span><span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">%role_ok</span> <span class="o">=</span> <span class="nb">map</span> <span class="p">{</span> <span class="nv">$_</span> <span class="o">=></span> <span class="mi">1</span> <span class="p">}</span> <span class="nv">@role_list</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="nv">user_roles</span><span class="p">())</span> <span class="p">{</span>
<span class="nv">$role_match</span><span class="o">++</span> <span class="ow">and</span> <span class="k">last</span> <span class="k">if</span> <span class="nv">$role_ok</span><span class="p">{</span><span class="nv">$_</span><span class="p">};</span>
<span class="p">}</span>
<span class="p">}</span> <span class="k">elsif</span> <span class="p">(</span><span class="nv">$mode</span> <span class="ow">eq</span> <span class="s">'all'</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$role_match</span><span class="o">++</span><span class="p">;</span>
<span class="k">for</span> <span class="k">my</span> <span class="nv">$role</span> <span class="p">(</span><span class="nv">@role_list</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nv">user_has_role</span><span class="p">(</span><span class="nv">$role</span><span class="p">))</span> <span class="p">{</span>
<span class="nv">$role_match</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">last</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$role_match</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nv">$coderef</span><span class="o">-></span><span class="p">();</span>
<span class="p">}</span>
<span class="nv">execute_hook</span><span class="p">(</span><span class="s">'permission_denied'</span><span class="p">,</span> <span class="nv">$coderef</span><span class="p">);</span>
<span class="k">return</span> <span class="nv">redirect</span> <span class="nv">$deniedpage</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">}</span>
<span class="nv">register_hook</span> <span class="sx">qw(login_required permission_denied)</span><span class="p">;</span>
</code></pre>
</div>
<p>主要摘要了几个部分:</p>
<ul>
<li>第一,register</li>
</ul>
<p>摘要中就是 register 了一个关键词 require_any_role 。这样在启用了本 plugin 的应用里,你可以直接使用这个关键词。至于具体的 sub,没有什么特殊的。看前面的用法举例就知道了,传递一个 roles 的数组引用(或者单个role的话就是字符串,这个在后面有判断)和一个 sub 作为参数,也就是 <code class="highlighter-rouge">@_</code>。</p>
<ul>
<li>第二,register_hook</li>
</ul>
<p>第一个是 <code class="highlighter-rouge">Dancer::Plugin</code> 的部分,第二个是 <code class="highlighter-rouge">Dancer::Hook</code> 的功能。注册一个叫 login_required 的 hook,然后在需要的地方运行 <code class="highlighter-rouge">execute_hook('login_required', $coderef)</code>。</p>
<p><code class="highlighter-rouge">register_hook</code> 接受 <code class="highlighter-rouge">$name</code> 和 <code class="highlighter-rouge">$coderef</code> 参数。如果只有 name 的话,<code class="highlighter-rouge">Dancer::Hook</code> 里也会自动生成一个 <code class="highlighter-rouge">$compiled_filter</code> ,作用就是除非你调用 <code class="highlighter-rouge">halt</code> 了,不然就输出一条 core 级别的日志(这里其实还用到了 <code class="highlighter-rouge">Dancer::Hook::Properties</code>,判断是否需要运行,默认初始化参数空的时候返回真,不运行 app,继续往下到记录日志)。然后,将这个对象传递给 <code class="highlighter-rouge">Dancer::Factory::Hook</code>。这里会把前面的生成的 coderef 加入到一个 <code class="highlighter-rouge">$class->hooks->{$hook_name}</code> 数组,而 name 加入到 <code class="highlighter-rouge">$self->registered_hooks</code> 数组。</p>
<p>在<code class="highlighter-rouge">execute_hook</code> 的时候,从前面的 <code class="highlighter-rouge">$self->registered_hooks</code> 判断是否有这个 name,然后从 <code class="highlighter-rouge">$class->hooks->{$hook_name}</code> 里依次取出全部 coderef 执行。</p>
<ul>
<li>第三,any</li>
</ul>
<p>和前面 0.001 类似,这里也有一个关键词冲突的问题,前面的 get 和这里的 any 都是 <code class="highlighter-rouge">Dancer</code> 的关键词。不然的话,其实这里使用 <code class="highlighter-rouge">Perl6::Junction</code> 或者 <code class="highlighter-rouge">Syntax::Keyword::Junction</code> 模块是正当其时啊。我之前都用 <code class="highlighter-rouge">Perl6::Junction</code>,不过昨天的 Perl Advent Calendar 文章里推荐了后面这个 <code class="highlighter-rouge">Syntax::Keyword::Junction</code>,<a href="https://metacpan.org">meta::cpan</a> 上也都是两个喜欢。另外题外话说一句,那篇文章里推荐的另一个 <a href="https://metacpan.org/module/Function::Parameters">Function::Parameters</a> 可真是好东西,唯一问题是低于 Perl 5.014的版本用不了,因为他不是 source filter 而是 keyword plugin api 的。这是新版本的功能。</p>
<hr />
<p><strong>12 月 30 日附:</strong></p>
<p>在 github 上提交了一个短短的 patch ,给 DPAE 加上了 正则匹配 role 的功能,感谢 Perl5.10的强大,代码其实就修改一行足以实现:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">lib</span><span class="sr">/Dancer/</span><span class="nv">Plugin</span><span class="sr">/Auth/</span><span class="nv">Extensible</span><span class="o">.</span><span class="nv">pm</span> <span class="nv">@</span> <span class="nv">891cd02</span>
<span class="nv">@@</span> <span class="err">-</span><span class="nv">266</span><span class="p">,</span><span class="mi">7</span> <span class="o">+</span><span class="mi">266</span><span class="p">,</span><span class="mi">9</span> <span class="nv">@@</span> <span class="nv">sub</span> <span class="nv">_build_wrapper</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$role_match</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$mode</span> <span class="ow">eq</span> <span class="s">'single'</span><span class="p">)</span> <span class="p">{</span>
<span class="o">-</span> <span class="nv">$role_match</span><span class="o">++</span> <span class="k">if</span> <span class="nv">user_has_role</span><span class="p">(</span><span class="nv">$require_role</span><span class="p">);</span>
<span class="o">+</span> <span class="k">for</span> <span class="p">(</span><span class="nv">user_roles</span><span class="p">())</span> <span class="p">{</span>
<span class="o">+</span> <span class="nv">$role_match</span><span class="o">++</span> <span class="ow">and</span> <span class="k">last</span> <span class="k">if</span> <span class="nv">$_</span> <span class="o">~~</span> <span class="nv">$require_role</span><span class="p">;</span>
<span class="o">+</span> <span class="p">}</span>
<span class="p">}</span> <span class="k">elsif</span> <span class="p">(</span><span class="nv">$mode</span> <span class="ow">eq</span> <span class="s">'any'</span><span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">%role_ok</span> <span class="o">=</span> <span class="nb">map</span> <span class="p">{</span> <span class="nv">$_</span> <span class="o">=></span> <span class="mi">1</span> <span class="p">}</span> <span class="nv">@role_list</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="nv">user_roles</span><span class="p">())</span> <span class="p">{</span>
<span class="nv">t</span><span class="o">/</span><span class="mo">01</span><span class="o">-</span><span class="nv">basic</span><span class="o">.</span><span class="nv">t</span> <span class="nv">@</span> <span class="nv">891cd02</span>
<span class="nv">@@</span> <span class="err">-</span><span class="nv">81</span><span class="p">,</span><span class="mi">6</span> <span class="o">+</span><span class="mi">81</span><span class="p">,</span><span class="mi">9</span> <span class="nv">@@</span> <span class="nv">response_status_is</span> <span class="p">[</span> <span class="nv">GET</span> <span class="o">=></span> <span class="s">'/allroles'</span> <span class="p">],</span> <span class="mi">200</span><span class="p">,</span>
<span class="nv">response_status_is</span> <span class="p">[</span> <span class="nv">GET</span> <span class="o">=></span> <span class="s">'/regex/a'</span> <span class="p">],</span> <span class="mi">200</span><span class="p">,</span>
<span class="s">"We can request a regex route when logged in"</span><span class="p">;</span>
<span class="o">+</span><span class="nv">response_status_is</span> <span class="p">[</span> <span class="nv">GET</span> <span class="o">=></span> <span class="s">'/piss/regex'</span> <span class="p">],</span> <span class="mi">200</span><span class="p">,</span>
<span class="o">+</span> <span class="s">"We can request a route requiring a regex role we have"</span><span class="p">;</span>
<span class="o">+</span>
<span class="c1"># ... but can't request something requiring a role we don't have</span>
<span class="nv">response_redirect_location_is</span> <span class="p">[</span> <span class="nv">GET</span> <span class="o">=></span> <span class="s">'/piss'</span> <span class="p">],</span>
<span class="s">'http://localhost/login/denied?return_url=%2Fpiss'</span><span class="p">,</span>
<span class="nv">t</span><span class="sr">/lib/</span><span class="nv">TestApp</span><span class="o">.</span><span class="nv">pm</span> <span class="nv">@</span> <span class="nv">891cd02</span>
<span class="nv">@@</span> <span class="err">-</span><span class="nv">39</span><span class="p">,</span><span class="mi">6</span> <span class="o">+</span><span class="mi">39</span><span class="p">,</span><span class="mi">10</span> <span class="nv">@@</span> <span class="nv">get</span> <span class="s">'/piss'</span> <span class="o">=></span> <span class="nv">require_role</span> <span class="nv">BearGrylls</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="s">"You can drink piss"</span><span class="p">;</span>
<span class="p">};</span>
<span class="o">+</span><span class="nv">get</span> <span class="s">'/piss/regex'</span> <span class="o">=></span> <span class="nv">require_role</span> <span class="sx">qr/beer/</span><span class="nv">i</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="o">+</span> <span class="s">"You can drink piss now"</span><span class="p">;</span>
<span class="o">+</span><span class="p">};</span>
<span class="o">+</span>
<span class="nv">get</span> <span class="s">'/anyrole'</span> <span class="o">=></span> <span class="nv">require_any_role</span> <span class="p">[</span><span class="s">'Foo'</span><span class="p">,</span><span class="s">'BeerDrinker'</span><span class="p">]</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="s">"Matching one of multiple roles works"</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
Dancer 框架使用 Text::XSlate 模版的注意事项
2012-12-19T00:00:00+08:00
dancer
http://chenlinux.com/2012/12/19/dancer-and-text-xslate
<p>Dancer 框架自带有一个 Simple 模版,不过推荐使用 <code class="highlighter-rouge">Template</code> 模块作为替代品。不过从性能上来说,TT2 比之前博客里陆续介绍过的 <code class="highlighter-rouge">HTML::Template</code> 和 <code class="highlighter-rouge">Text::MicroTemplate</code> 都要差。而这方面最好的,就是 <code class="highlighter-rouge">Text::XSlate</code> 模块了。今天尝试将一个 Dancer 应用迁移到 <code class="highlighter-rouge">Text::XSlate</code> 上。踩进两个坑,特此记录。</p>
<p>关于语法什么的,可以看 POD ,扶凯 有翻译的 中文版POD 。足以十分钟入门。就不多说了。</p>
<ul>
<li>第一个坑:session 的处理</li>
</ul>
<p>website 少不了 session 的运用。在 template 里使用 <code class="highlighter-rouge">[% session.username %]</code> 可以很方便的控制显示面板还是登陆啊什么的。</p>
<p>不过切换成 XSlate 后(即 <code class="highlighter-rouge"><: $session.username :></code>),请求会 crash 掉,报错大意是: <strong>$session 没有 username 这个 method</strong>。</p>
<p>XSlate 提供了 <code class="highlighter-rouge">dump</code> 语法糖,让我们可以直接使用 <code class="highlighter-rouge"><: $session | dump :></code> 检查问题。这时候发现显示如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">$VAR1</span> <span class="o">=</span> <span class="p">{</span> <span class="nv">blessed</span><span class="p">(</span> <span class="p">{</span> <span class="nv">id</span> <span class="o">=></span> <span class="s">'2131232131'</span><span class="p">,</span> <span class="nv">username</span> <span class="o">=></span> <span class="s">'user1'</span> <span class="p">}</span> <span class="p">),</span> <span class="nn">Dancer::Session::</span><span class="nv">YAML</span> <span class="p">};</span>
</code></pre>
</div>
<p>尝试使用 <code class="highlighter-rouge"><: $session.id :></code> ,发现可以正常输出 2131232131 。</p>
<p>进去看 <code class="highlighter-rouge">Dancer::Session</code> 的代码,原来在 <code class="highlighter-rouge">Dancer::Session::Abstract</code> 里,有这么一行:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">__PACKAGE__</span><span class="o">-></span><span class="nv">attributes</span><span class="p">(</span><span class="s">'id'</span><span class="p">);</span>
</code></pre>
</div>
<p>说实话不太理解这行的用法,不过不妨碍我们用简单办法解决问题…… 在我们的应用中给 <code class="highlighter-rouge">Dancer::Session::YAML</code> 定义一个叫 username 的 method 就可以骗过去了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">package</span> <span class="nv">DancerApp</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Dancer</span> <span class="s">':syntax'</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Dancer::Session::</span><span class="nv">YAML</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">Dancer</span><span class="p">::Session::YAML::username {</span>
<span class="k">return</span> <span class="nv">session</span><span class="p">(</span><span class="s">'username'</span><span class="p">);</span>
<span class="p">};</span>
<span class="k">use</span> <span class="nn">Dancer::Plugin::Auth::</span><span class="nv">Extensible</span><span class="p">;</span>
<span class="nv">get</span> <span class="s">'/'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">:RequireLogin {</span> <span class="nv">template</span> <span class="s">'index'</span> <span class="p">};</span>
<span class="o">...</span><span class="p">;</span>
<span class="nv">true</span><span class="p">;</span>
</code></pre>
</div>
<p><strong>2013 年 03 月 25 日更新</strong></p>
<p>今天莫莫也换成 Xslate 模板,顺带告诉我这里一个更通用和优雅的修改方式:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">package</span> <span class="nv">DancerApp</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Dancer</span> <span class="s">':syntax'</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Dancer::Session::</span><span class="nv">Abstract</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Dancer::Plugin::Auth::</span><span class="nv">Extensible</span><span class="p">;</span>
<span class="nn">Dancer::Session::</span><span class="nv">Abstract</span><span class="o">-></span><span class="nv">attributes</span><span class="p">(</span> <span class="sx">qw(username)</span> <span class="p">);</span>
<span class="nv">get</span> <span class="s">'/'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">:RequireLogin {</span> <span class="nv">template</span> <span class="s">'index'</span> <span class="p">};</span>
<span class="o">...</span><span class="p">;</span>
<span class="nv">true</span><span class="p">;</span>
</code></pre>
</div>
<p>这样可以在各种 Session 引擎下通用了。</p>
<p><strong>更新完毕</strong></p>
<ul>
<li>第二个坑:flashmessage 的处理</li>
</ul>
<p>这是一个外加模块,叫做 <code class="highlighter-rouge">Dancer::Plugin::FlashMessage</code> 。用它配合模版的 layout 功能,可以很方便的给应用提供全局的消息通知。使用方法如下:</p>
<p>首先在模块里加载 flash 变量:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">package</span> <span class="nn">DancerApp::</span><span class="nv">First</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Dancer</span> <span class="s">':syntax'</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Dancer::Plugin::</span><span class="nv">FlashMessage</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Dancer::Plugin::Auth::</span><span class="nv">Extensible</span><span class="p">;</span>
<span class="nv">get</span> <span class="s">'/first/:name'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">:RequireRole('MAN') {</span>
<span class="nv">flash</span> <span class="nv">message</span> <span class="o">=></span> <span class="s">'Hello! You are the first man here.'</span><span class="p">;</span>
<span class="nv">template</span> <span class="s">'first'</span><span class="p">,</span> <span class="p">{</span> <span class="nv">name</span> <span class="o">=></span> <span class="nv">param</span><span class="p">(</span><span class="s">'name'</span><span class="p">)</span> <span class="p">};</span>
<span class="p">};</span>
<span class="nv">true</span><span class="p">;</span>
</code></pre>
</div>
<p>然后在模版里判断显示:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> [% IF flash.message %]
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"alert alert-success"</span><span class="nt">></span>
[% flash.message %]
<span class="nt"></div></span>
[% END %]
</code></pre>
</div>
<p>同样,在修改成 XSlate 后,模版是这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> : if $flash.message {
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"alert alert-success"</span><span class="nt">></span>
<span class="nt"><:</span> <span class="na">flash</span><span class="err">.</span><span class="na">message</span> <span class="na">:</span><span class="nt">></span>
<span class="nt"></div></span>
: }
</code></pre>
</div>
<p>结果发现页面上的 div 一直保持,而且显示着 <code class="highlighter-rouge">CODE(0x39a5c30)</code> 这样的字样。同样使用 <code class="highlighter-rouge">dump</code> 语法糖,看到 <code class="highlighter-rouge">$flash</code> 其实是 <code class="highlighter-rouge"><span class="p">{</span><span class="w"> </span><span class="err">message</span><span class="w"> </span><span class="err">=></span><span class="w"> </span><span class="err">sub</span><span class="w"> </span><span class="err">{</span><span class="nt">"DUMMY"</span><span class="err">}</span><span class="w"> </span><span class="err">}</span></code>。</p>
<p>这个就有趣了,居然是个代码段~~于是翻源码来看:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">hook</span> <span class="nv">before_template</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="nb">shift</span><span class="o">-></span><span class="p">{</span><span class="nv">$token_name</span><span class="p">}</span> <span class="o">=</span> <span class="p">{</span>
<span class="nb">map</span> <span class="p">{</span> <span class="k">my</span> <span class="nv">$key</span> <span class="o">=</span> <span class="nv">$_</span><span class="p">;</span> <span class="k">my</span> <span class="nv">$value</span><span class="p">;</span>
<span class="p">(</span> <span class="nv">$key</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span> <span class="nb">defined</span> <span class="nv">$value</span> <span class="ow">and</span> <span class="k">return</span> <span class="nv">$value</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$flash</span> <span class="o">=</span> <span class="nv">session</span><span class="p">(</span><span class="nv">$session_hash_key</span><span class="p">)</span> <span class="o">||</span> <span class="p">{};</span>
<span class="nv">$value</span> <span class="o">=</span> <span class="nb">delete</span> <span class="nv">$flash</span><span class="o">-></span><span class="p">{</span><span class="nv">$key</span><span class="p">};</span>
<span class="nv">session</span> <span class="nv">$session_hash_key</span><span class="p">,</span> <span class="nv">$flash</span><span class="p">;</span>
<span class="k">return</span> <span class="nv">$value</span><span class="p">;</span>
<span class="p">}</span> <span class="p">);</span>
<span class="p">}</span> <span class="p">(</span> <span class="nb">keys</span> <span class="nv">%</span><span class="p">{</span><span class="nv">session</span><span class="p">(</span><span class="nv">$session_hash_key</span><span class="p">)</span> <span class="o">||</span> <span class="p">{}</span> <span class="p">})</span>
<span class="p">};</span>
<span class="p">};</span>
</code></pre>
</div>
<p><code class="highlighter-rouge">map</code> 里面,确实是一个 <code class="highlighter-rouge">$key => sub {}</code> 。</p>
<p>这个时候我切换两个 template 做了个测试。在里面那个匿名 sub 里写了一行 <code class="highlighter-rouge">die;</code>。结果。XSlate “正常”运行过去,在页面上显示前面说过的 <code class="highlighter-rouge">CODE()</code>;而在 Template 模版下,500 了。看 console 的日志,发现 <code class="highlighter-rouge">die</code> 这个动作不是在 <code class="highlighter-rouge">before_template</code> 阶段发生的。而是在随后的 render 阶段,<code class="highlighter-rouge">Dancer::Template::Abstract</code> 里才挂了。</p>
<p>所以,最终,两个坑归结起来并成了一个问题:模版系统是支持 coderef 还是支持 object 的问题。就在我写着这句话的同时,IRC 上还为 <code class="highlighter-rouge">Dancer::Plugin::FlashMessage</code> 的新实现而争论不休。xdg 童鞋已经在我提问的一个小时内快速的搞出来一个把 flash “object 化”的 patch,而 bigpresh 童鞋坚定的认为应该把 <code class="highlighter-rouge">delete</code> 操作放在 <code class="highlighter-rouge">hook after_template</code> 里完成。原作者 <code class="highlighter-rouge">dams</code> 则”相信”更多的模版是支持 coderef 不支持 object 的。</p>
<p>不过我觉得,其实改动最小的办法,就是别用 <code class="highlighter-rouge">map</code> 这么高档的语法。拆成两段处理,确保传递给 <code class="highlighter-rouge">template_render</code> 的是字符串即可:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">hook</span> <span class="nv">before_template</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">%hash</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$flash</span> <span class="o">=</span> <span class="nv">session</span><span class="p">(</span><span class="nv">$session_hash_key</span><span class="p">)</span> <span class="o">||</span> <span class="p">{};</span>
<span class="k">for</span> <span class="p">(</span> <span class="nb">keys</span> <span class="nv">%</span><span class="p">{</span><span class="nv">$flash</span><span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$hash</span><span class="p">{</span><span class="nv">$_</span><span class="p">}</span> <span class="o">=</span> <span class="nb">delete</span> <span class="nv">$flash</span><span class="o">-></span><span class="p">{</span><span class="nv">$_</span><span class="p">};</span>
<span class="p">};</span>
<span class="nv">session</span> <span class="nv">$session_hash_key</span><span class="p">,</span> <span class="nv">$flash</span><span class="p">;</span>
<span class="nb">shift</span><span class="o">-></span><span class="p">{</span><span class="nv">$token_name</span><span class="p">}</span> <span class="o">=</span> <span class="o">\</span><span class="nv">%hash</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<p>最后的最后,就在我测试完我的改动版本在两种模版下都可以运行的时候,dams 已经决定先同时保持 coderef 和 object 的写法并提供 setting 配置。然后慢慢搜集各种模版系统做覆盖测试。</p>
<p><strong>20 日增</strong></p>
<p>最后最后的最后,在 github 上搜到两个用 dancer 和 xslate 写的 repo。他们都采用了在应用 app 里自定义 <code class="highlighter-rouge">hook before_template</code> ,把 <code class="highlighter-rouge">session('username')</code> 和 <code class="highlighter-rouge">flash('message')</code> 两个变量传递给 <code class="highlighter-rouge">$token</code> 哈希的办法。</p>
perl发起HTTP请求时如何设置Host头
2012-12-16T00:00:00+08:00
perl
http://chenlinux.com/2012/12/16/how-to-set-host-header-in-perl
<p>之所以写这么个内容,是今天突然发现之前有个脚本的效果完全不对。这个脚本是用 Furl 模块发 HTTP 请求。看 POD 的说明,以为这样写是生效的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nn">HTTP::</span><span class="nv">Request</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Furl</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$r</span> <span class="o">=</span> <span class="nn">HTTP::</span><span class="nv">Request</span><span class="o">-></span><span class="k">new</span><span class="p">(</span> <span class="nv">GET</span> <span class="o">=></span> <span class="s">"http://192.168.0.2/path/to/file"</span> <span class="p">);</span>
<span class="nv">$r</span><span class="o">-></span><span class="nv">header</span><span class="p">(</span> <span class="nv">Host</span> <span class="o">=></span> <span class="s">"www.example.com"</span> <span class="p">);</span>
<span class="k">my</span> <span class="nv">$furl</span> <span class="o">=</span> <span class="nv">Furl</span><span class="o">-></span><span class="k">new</span><span class="p">();</span>
<span class="k">my</span> <span class="nv">$res</span> <span class="o">=</span> <span class="nv">$furl</span><span class="o">-></span><span class="nv">request</span><span class="p">(</span><span class="nv">$r</span><span class="p">);</span>
<span class="nv">say</span> <span class="nv">$res</span><span class="o">-></span><span class="nv">code</span><span class="p">();</span>
</code></pre>
</div>
<p>但是随后在 192.168.0.2 上发现日志记录中,Host 并没有修改成 www.example.com 。</p>
<p>然后尝试了各种 POD 上介绍的 header 写法,包括在 new HTTP::Request 的时候使用 <code class="highlighter-rouge">[Host => "www.example.com"]</code> 参数,在 <code class="highlighter-rouge">$furl->request</code> 的时候使用 <code class="highlighter-rouge">headers => [Host => "www.example.com"]</code> 参数。结果都一样。</p>
<p>然后只能改思路,用设置 proxy 的办法。结果发现 Furl 模块的 proxy 不可用……</p>
<p>POD 上是说直接在 new 的时候传递 %args 或者 \%args 就行。但是我使用的时候发现直接会报错:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Passed malformed URL: 192.168.0.2
</code></pre>
</div>
<p>最后只能放弃使用 Furl 模块,改回古老的 LWP 模块。LWP 与 Coro 配合如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nv">Coro</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">LWP::Protocol::Coro::</span><span class="nv">http</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">LWP::</span><span class="nv">UserAgent</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">co_http_get</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$domain</span><span class="p">,</span> <span class="nv">$urlpath</span><span class="p">,</span> <span class="nv">$iplist</span> <span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@coros</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$msg</span> <span class="o">=</span> <span class="s">''</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$ua</span> <span class="o">=</span> <span class="nn">LWP::</span><span class="nv">UserAgent</span><span class="o">-></span><span class="k">new</span><span class="p">();</span>
<span class="k">foreach</span> <span class="k">my</span> <span class="nv">$ip</span> <span class="p">(</span> <span class="nv">@</span><span class="p">{</span><span class="nv">$iplist</span><span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="nb">push</span> <span class="nv">@coros</span><span class="p">,</span> <span class="nv">async</span> <span class="p">{</span>
<span class="nv">$ua</span><span class="o">-></span><span class="nv">proxy</span><span class="p">(</span><span class="s">'http'</span><span class="p">,</span> <span class="s">"http://$ip:3128/"</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$res</span> <span class="o">=</span> <span class="nv">$ua</span><span class="o">-></span><span class="nv">get</span><span class="p">(</span><span class="s">"http://$domain$urlpath"</span><span class="p">);</span>
<span class="nv">$msg</span> <span class="o">.=</span> <span class="s">"$ip: "</span> <span class="o">.</span> <span class="nv">$res</span><span class="o">-></span><span class="nv">code</span><span class="p">()</span> <span class="o">.</span> <span class="s">"\n"</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nv">$_</span><span class="o">-></span><span class="nb">join</span> <span class="k">for</span> <span class="nv">@coros</span><span class="p">;</span>
<span class="k">return</span> <span class="nv">$msg</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">print</span> <span class="nv">co_http_get</span><span class="p">(</span><span class="s">"www.example.com"</span><span class="p">,</span> <span class="s">"/path/to/file"</span><span class="p">,</span> <span class="p">[</span><span class="sx">qw(192.168.0.1 192.168.0.2)</span><span class="p">]);</span>
</code></pre>
</div>
不小心踩进ElasticSearch.pm模块的坑里了
2012-12-11T00:00:00+08:00
logstash
elasticsearch
perl
http://chenlinux.com/2012/12/11/note-on-using-ElasticSearch-pm
<p>在今天以前,我一直认为perl的ElasticSearch.pm是除了原生java库以外封装最好的。不过今天踩进一个硕大的坑里,多亏 dancer-user 邮件列表里外国友人的帮助,才算爬了出来……</p>
<h1 id="section">事情是这样的</h1>
<p>用 dancer 搭建的一个 webserver 用来提供 api 给前端图表页面。dancer 收到 ajax 请求后组装成 json 发给 ElasticSearch。因为要算百分比,无法在单次请求内完成,不然的话直接从页面上发给 ES 服务器了。</p>
<p>这个 webserver 是之前已经创建过的。而且作用类似,也就是说,之前已经存在一个 <code class="highlighter-rouge">DancerApp/lib/DancerApp/First.pm</code> 里使用了 ElasticSearch 模块。相关代码如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nv">Dancer</span> <span class="s">':syntax'</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">ElasticSearch</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$elsearch</span> <span class="o">=</span> <span class="nv">ElasticSearch</span><span class="o">-></span><span class="k">new</span><span class="p">(</span> <span class="nv">config</span><span class="o">-></span><span class="p">{</span><span class="nv">ElasticSearch</span><span class="p">}</span> <span class="p">);</span>
</code></pre>
</div>
<p>然后给新项目创建 <code class="highlighter-rouge">DancerApp/lib/DancerApp/Second.pm</code> 同样使用 ElasticSearch 模块,代码原样复制。然后在 <code class="highlighter-rouge">DancerApp/lib/DancerApp.pm</code> 里先后加载:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nv">Dancer</span> <span class="s">':syntax'</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">FindBin</span> <span class="sx">qw($Bin)</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">lib</span> <span class="s">"$Bin/../lib"</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">DancerApp::</span><span class="nv">First</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">DancerApp::</span><span class="nv">Second</span><span class="p">;</span>
</code></pre>
</div>
<p>启动应用后访问页面。怪事出现了: <em>First 应用正常,Second 应用报错说 ElasticSearch 连接不上</em>。</p>
<p>仔细看报错信息,发现Second 里的 <code class="highlighter-rouge">$elsearch</code> 连接的不是 <code class="highlighter-rouge">config.yml</code> 里设定的 servers,而是模块默认的 <code class="highlighter-rouge">127.0.0.1:9200</code>。</p>
<p>更换<code class="highlighter-rouge">DancerApp/lib/DancerApp.pm</code> 里的加载次序,就变成了 <em>Second 正常,First 失败</em>。</p>
<p>试图使用下面的代码检查 <code class="highlighter-rouge">config</code> ,发现 config 里其他的设置都没问题,唯独和 ElasticSearch 相关的设定发生了变化:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nn">Data::</span><span class="nv">Dumper</span><span class="p">;</span>
<span class="nv">get</span> <span class="s">'/config'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="k">return</span> <span class="nv">Dumper</span> <span class="nv">config</span> <span class="p">};</span>
</code></pre>
</div>
<p>结果中 <code class="highlighter-rouge">config->{ElasticSearch}</code> 只剩下 <code class="highlighter-rouge">trace_calls: 0</code> 一条设定, <code class="highlighter-rouge">servers</code>、<code class="highlighter-rouge">transport</code>、<code class="highlighter-rouge">no_refresh</code> 和 <code class="highlighter-rouge">max_requests</code> 都消失了!</p>
<h1 id="section-1">真相只有一个</h1>
<p>ElasticSearch 模块在初始化的时候,会把参数传递给 <code class="highlighter-rouge">ElasticSearch::Transport</code> 模块做具体的操作(包括之前我很欣赏的自动选择节点服务器)。而就在这里,问题出现了:</p>
<p><em>参数一直是以引用身份传递的,任何修改都会修改原始数据</em></p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">my</span> <span class="nv">$servers</span> <span class="o">=</span> <span class="nb">delete</span> <span class="nv">$params</span><span class="o">-></span><span class="p">{</span><span class="nv">servers</span><span class="p">}</span>
<span class="o">||</span> <span class="s">'127.0.0.1:'</span> <span class="o">.</span> <span class="nv">$transport_class</span><span class="o">-></span><span class="nv">default_port</span><span class="p">;</span>
</code></pre>
</div>
<p>随着 <code class="highlighter-rouge">delete</code> 操作,悲剧就此发生了。Dancer 里的全局变量 <code class="highlighter-rouge">config->{ElasticSearch}</code> 中的 servers 元素就此消失……</p>
<h1 id="section-2">善后事宜</h1>
<p>解决办法很容易,在每个模块里初始化 ElasticSearch 实例的适合,传递一个全局 <code class="highlighter-rouge">config->{ElasticSearch}</code> 的_副本的引用_过去。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">my</span> <span class="nv">$elsearch</span> <span class="o">=</span> <span class="nv">ElasticSearch</span><span class="o">-></span><span class="k">new</span><span class="p">(</span> <span class="p">{</span> <span class="nv">%</span><span class="p">{</span> <span class="nv">config</span><span class="o">-></span><span class="p">{</span><span class="nv">ElasticSearch</span><span class="p">}</span> <span class="p">}</span> <span class="p">}</span> <span class="p">);</span>
</code></pre>
</div>
<p>亲爱的 David Precious 童鞋已经把这个问题上报给 ElasticSearch.pm 开发者了。或许之后会由模块内部做副本操作。目前只能自己来了。</p>
<p>issue 地址:<a href="https://github.com/clintongormley/ElasticSearch.pm/issues/34">https://github.com/clintongormley/ElasticSearch.pm/issues/34</a></p>
用gnuplot绘制直方图
2012-12-10T00:00:00+08:00
monitor
gnuplot
http://chenlinux.com/2012/12/10/gnuplot-to-draw-histogram-cluster
<p>越来越喜欢用 gnuplot 画图了,因为有时候发现自己实在是不会用 Excel……</p>
<p>之前基本上用gnuplot画的都是时间轴形式的,诸位客官肯定已经看多了。但是 gnuplot 可不止是这点。还有一种很常见的功能也很方便,就是与时间无关的多个数据做对比的时候,还画成两条连线,就不如画成直方图更体现价值了。比如下面这组数据:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>地区 总数(A) 总数(B)
美洲 16682 20344
澳洲 4021 3672
欧洲 2902 2878
</code></pre>
</div>
<p>只需要几行配置,就可以生成很漂亮滴直方图对比了。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nb">set </span>key right top Left reverse width 0 box 3
<span class="nb">set </span>xlabel <span class="s2">"各大洲区域"</span>
<span class="nb">set </span>ylabel <span class="s2">"请求总数"</span>
<span class="nb">set </span>ytics 0, 500
<span class="nb">set </span>mytics 5
<span class="nb">set </span>grid
<span class="nb">set </span>boxwidth 0.9 absolute
<span class="nb">set </span>style fill solid 1.00 border -1
<span class="nb">set </span>style histogram clustered gap 1 title offset character 0, 0, 0
<span class="nb">set </span>style data histograms
<span class="nb">set </span>terminal png size 1024, 512
<span class="nb">set </span>output <span class="s2">"oversea.png"</span>
plot <span class="s1">'oversea.csv'</span> using 2:xtic<span class="o">(</span>1<span class="o">)</span> ti col, <span class="s1">''</span> u 3 ti col
</code></pre>
</div>
<p>注意如果行比较多,默认大小的图上X轴的标记就会挤在一块了,所以在 set terminal 后面设置图片大小,这和 set size 是不一样的。后者设置的相对值是本次要 plot 的图形在总画布上的比例大小。</p>
<p>plot 里 using 的两列也是和画 line 图时反过来的顺序,而且 X 轴的列要用 xtic() 包起来写,否则 gnuplot 会认为这应该是个自增序列,然后找不到 xrange 出错。</p>
<p>效果图如下:</p>
<p><img src="/images/uploads/gnuplot-boxes.png" alt="图片" /></p>
把docx文档转换成markdown格式发布
2012-12-01T00:00:00+08:00
ruby
http://chenlinux.com/2012/12/01/convert-docx-to-markdown
<p>有些Word文档想搬到博客上来,而博客用的是markdown的格式。最简单的办法是在Word里转成html格式另存为,因为markdown和html是兼容的。不过word直接另存为的html里面带有“海量”的无聊样式,实在不方便之后我们再用vim的工具编辑。所以还是想办法整整。</p>
<p>相对来说,Word的docx格式比doc格式要容易处理,因为docx是微软特意推出的open xml格式。其实就是记录了文本内容的content.xml、附件media/*和对应附件路径的_ref.xml等的zip包而已。所以相对必须在Windows平台上调用WIN32OLE的API来处理的doc来说,我们在linux平台上也可以很容易的处理docx文件了。比如rubygems上就有一个很不错的gem叫<code class="highlighter-rouge">ydocx</code>。一般的docx库都是只抽取docx里的content文字,而这个ydocx很负责的把media/*也复制到docxname_files/images/*下面,并且在html里生成<code class="highlighter-rouge"><img></code>标签了。</p>
<p>然后另一步就是把html转换成markdown,这在github上也有现成的repo叫<a href="https://github.com/cousine/downmark_it">downmark_it</a>。嗯,这名字一目了然就是反过来……</p>
<p>(<code class="highlighter-rouge">ydocx</code>用的是<code class="highlighter-rouge">nokogiri</code>,<code class="highlighter-rouge">downmark\_it</code>用的是<code class="highlighter-rouge">hpricot</code>,或许应该也改用<code class="highlighter-rouge">nokogiri</code>比较好~不过<code class="highlighter-rouge">nokogiri</code>官网可耻的被墙了)</p>
<h1 id="section">首先安装依赖</h1>
<div class="highlighter-rouge"><pre class="highlight"><code> apt-get install libxslt1-dev libxml2-dev
gem install rubyzip htmlentities rmagick ydocx hpricot
wget https://raw.github.com/cousine/downmark_it/master/downmark_it.rb
</code></pre>
</div>
<h1 id="section-1">编写转换脚本</h1>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">require</span> <span class="s1">'rubygems'</span>
<span class="nb">require</span> <span class="s1">'ydocx'</span>
<span class="vg">$:</span> <span class="o"><<</span> <span class="no">File</span><span class="p">.</span><span class="nf">dirname</span><span class="p">(</span><span class="kp">__FILE__</span><span class="p">)</span>
<span class="nb">require</span> <span class="s1">'downmark_it'</span>
<span class="n">filename</span> <span class="o">=</span> <span class="no">ARGV</span><span class="p">.</span><span class="nf">shift</span>
<span class="n">ydocx</span> <span class="o">=</span> <span class="no">YDocx</span><span class="o">::</span><span class="no">Document</span><span class="p">.</span><span class="nf">open</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="n">html</span> <span class="o">=</span> <span class="n">ydocx</span><span class="p">.</span><span class="nf">to_html</span><span class="p">.</span><span class="nf">gsub</span><span class="p">(</span><span class="sr">/\n/</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
<span class="nb">puts</span> <span class="no">DownmarkIt</span><span class="p">.</span><span class="nf">to_markdown</span><span class="p">(</span><span class="n">html</span><span class="p">)</span>
</code></pre>
</div>
<p>这样就能看到输出了。目录里的每个章节都有引用格式凸现,美中不足是对word里的标题样式识别不太好,本来期望是可以自己生成<code class="highlighter-rouge"><h1></code>、<code class="highlighter-rouge"><h2></code>的,但是ydocx生成的html里只把第一个标题一变成<code class="highlighter-rouge"><h1></code>,其他的都是普通的<code class="highlighter-rouge"><p></code>。</p>
<p>另一个问题是上面脚本里直接调用to_html的方法,不会保存住unzip出来的images文件夹。自己再另写一段unzip的代码:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">require</span> <span class="s1">'fileutils'</span>
<span class="nb">require</span> <span class="s1">'zip/zip'</span>
<span class="nb">require</span> <span class="s1">'zip/zipfilesystem'</span>
<span class="k">def</span> <span class="nf">unzip</span><span class="p">(</span><span class="n">zip_file</span><span class="p">,</span> <span class="n">dest_dir</span><span class="p">)</span>
<span class="no">Zip</span><span class="o">::</span><span class="no">ZipFile</span><span class="p">.</span><span class="nf">open</span><span class="p">(</span><span class="n">zip_file</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">zf</span><span class="o">|</span>
<span class="n">zf</span><span class="p">.</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">e</span><span class="o">|</span>
<span class="n">path</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">dest_dir</span><span class="p">,</span> <span class="n">e</span><span class="p">.</span><span class="nf">name</span><span class="p">)</span>
<span class="no">FileUtils</span><span class="p">.</span><span class="nf">mkdir_p</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">dirname</span><span class="p">(</span><span class="n">path</span><span class="p">))</span>
<span class="n">zf</span><span class="p">.</span><span class="nf">extract</span><span class="p">(</span><span class="n">e</span><span class="p">,</span> <span class="n">path</span><span class="p">)</span> <span class="p">{</span> <span class="kp">true</span> <span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">dirname</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">basename</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">'.docx'</span><span class="p">)</span>
<span class="n">unzip</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s2">"/tmp/</span><span class="si">#{</span><span class="n">dirname</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="no">FileUtils</span><span class="p">.</span><span class="nf">mv</span><span class="p">(</span><span class="s2">"/tmp/</span><span class="si">#{</span><span class="n">dirname</span><span class="si">}</span><span class="s2">/media/"</span><span class="p">,</span> <span class="s2">"/images/"</span><span class="p">)</span>
<span class="no">FileUtils</span><span class="p">.</span><span class="nf">rm_rf</span><span class="p">(</span><span class="s2">"/tmp/</span><span class="si">#{</span><span class="n">dirname</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
</code></pre>
</div>
<p>比较普通的办法,是直接使用ydocx自带的脚本<code class="highlighter-rouge">docx2html --format none file.docx</code>,会在docx文档的同级目录下生成同名html和_files目录。然后再写一个单行脚本转成markdown的。</p>
用 Tatsumaki 框架写 elasticsearch 界面
2012-11-22T00:00:00+08:00
logstash
perl
javascript
amcharts
http://chenlinux.com/2012/11/22/tatsumaki-demo
<p>Tatsumaki是Plack作者的一个小框架,亮点是很好的利用了psgi.streaming的接口可以async的完成响应。不过因为缺少周边支持,所以除了几个webchat的example,似乎没看到什么应用。笔者之前纯为练手,却用tatsumaki写了个sync响应的小demo,算是展示一下用tatsuamki做普通web应用的基础步骤吧:</p>
<p>(代码本来是作为一个ElasticSearch数据分析的平台,不过后来发现社区有人开始做纯js的内嵌进ElasticSearch的plugin了,所以撤了repo,这里贴下代码)</p>
<ul>
<li>
<p>所有的psgi/plack应用都一样有自己的app.psgi文件:<br />
```perl<br />
our $VERSION = 0.01;<br />
### app.psgi<br />
use Tatsumaki::Error;<br />
use Tatsumaki::Application;<br />
use Tatsumaki::HTTPClient;<br />
use Tatsumaki::Server;<br />
### read config<br />
use File::Basename;<br />
use YAML::Syck;<br />
my $config = LoadFile(dirname(<strong>FILE</strong>) . ‘/config.yml’);<br />
### elasticsearch init<br />
use ElasticSearch;<br />
#这里yml的写法借鉴Dancer::Plugin::ElasticSearch了<br />
my $elsearch = ElasticSearch->new( $config->{‘options’} );<br />
### index init<br />
use POSIX qw(strftime);<br />
my $index = join ‘-‘, ( (+split( ‘-‘, $config->{‘index’} ))[0], strftime( (+split( ‘-‘, $config->{‘index’} ))[1], localtime ) );<br />
my $type = $config->{‘type’};<br />
#首页类,调用了模板<br />
package MainHandler;<br />
use parent qw(Tatsumaki::Handler);<br />
sub get {<br />
my $self = shift;<br />
$self->render(‘index.html’);<br />
};<br />
#具体的API类<br />
package ListHandler;<br />
use parent qw(Tatsumaki::Handler);<br />
sub get {<br />
#这里自动把urlpath切分好了<br />
my ( $self, $group, $order, $interval ) = @_;<br />
return ‘Not valid order’ unless $order eq ‘count’ or $order eq ‘mean’;<br />
return ‘Not valid interval’ unless $interval =~ m#\d+(h|m|s)#;<br />
my ($key_field, $value_field);<br />
if ( $group eq ‘url’ ) {<br />
$key_field = ‘url’;<br />
$value_field = ‘responsetime’;<br />
} elsif ( $group eq ‘ip’ ) {<br />
$key_field = ‘oh’;<br />
$value_field = ‘upstreamtime’;<br />
} else {<br />
return ‘Not valid group field’;<br />
};</p>
<p># get index mapping and sort into array<br />
my $mapping = $elsearch->mapping(<br />
index => “$index”,<br />
type => “$type”,<br />
);<br />
my @res_map;<br />
for my $property ( sort keys %{ $mapping->{$type}->{‘properties’} } ) {<br />
if ($property eq ‘@fields’ ) {<br />
my @fields;<br />
push @fields, { name => $<em>, type => $mapping->{$type}->{‘properties’}->{$property}->{‘properties’}->{$</em>}->{‘type’} }<br />
for sort keys %{ $mapping->{$type}->{‘properties’}->{$property}->{‘properties’} };<br />
push @res_map, \@fields;<br />
} else {<br />
push @res_map, { name => $property, type => $mapping->{$type}->{‘properties’}->{$property}->{‘type’} };<br />
}<br />
}</p>
<p># get value stat group by key field<br />
my $data = $elsearch->search(<br />
index => “$index”,<br />
type => “$type”,<br />
size => 0,<br />
query => {<br />
“range” => {<br />
‘@timestamp’ => {<br />
from => “now-$interval”,<br />
to => “now”<br />
},<br />
},<br />
},<br />
facets => {<br />
“$group” => {<br />
“terms_stats” => {<br />
“value_field” => “$value_field”,<br />
“key_field” => “$key_field”,<br />
“order” => “$order”,<br />
“size” => 20,<br />
}<br />
},<br />
}<br />
);<br />
my @res_tbl;<br />
for ( @{$data->{facets}->{“$group”}->{terms}} ) {<br />
my $key = $<em>->{term};<br />
my $mean = sprintf “%.03f”, $</em>->{mean};<br />
my $code_count = code_count($key_field, $key, $interval);<br />
push @res_tbl, {<br />
key => $key,<br />
min => $<em>->{min},<br />
max => $</em>->{max},<br />
mean => $mean,<br />
code => $code_count,<br />
count => $_->{count},<br />
};<br />
};</p>
</li>
</ul>
<h1 id="renderselfkeyhandler">render可以接收参数,并且默认把$self带进去,具体key是handler</h1>
<div class="highlighter-rouge"><pre class="highlight"><code>$self->render('index.html', { table => \@res_tbl, mapping => \@res_map }); };
</code></pre>
</div>
<p>sub code_count {<br />
my ($key_field, $key, $interval) = @<em>;<br />
my $result;<br />
my $data = $elsearch->search(<br />
index => “$index”,<br />
type => “$type”,<br />
size => 0,<br />
query => {<br />
range => {<br />
‘@timestamp’ => {<br />
from => “now-$interval”,<br />
to => “now”<br />
},<br />
},<br />
},<br />
facets => {<br />
“code” => {<br />
facet_filter => {<br />
term => {<br />
$key_field => “$key”<br />
}<br />
},<br />
terms => {<br />
field => “status”,<br />
}<br />
}<br />
}<br />
);<br />
for ( @{$data->{facets}->{code}->{terms}} ) {<br />
$result->{$</em>->{term}} = $_->{count};<br />
};<br />
return $result;<br />
};<br />
#画图数据API类,因为响应的是Ajax请求,所以这里开启了async,不过其实没意义了。因为这个ElasticSearch代码不是async格式的。应该改造用ElasticSearch::Transport::AEHTTP才能做到全程async。<br />
package ChartHandler;<br />
use parent qw(Tatsumaki::Handler);<br />
<strong>PACKAGE</strong>->asynchronous(1);<br />
use JSON;<br />
sub post {<br />
my $self = shift;<br />
my $api = $self->request->param(‘api’) || ‘term’;<br />
my $key = $self->request->param(‘key’) || ‘oh’;<br />
my $value = $self->request->param(‘value’);<br />
my $status = $self->request->param(‘status’) || ‘200’;</p>
<div class="highlighter-rouge"><pre class="highlight"><code>my $field = $key eq 'oh' ? 'upstreamtime' : 'responsetime';
my $data = $elsearch->search(
index => "$index",
type => "$type",
size => 0,
query => {
match_all => { }
},
facets => {
"chart" => {
facet_filter => {
and => [
{
term => {
status => $status,
},
},
{
$api => {
$key => $value,
},
},
]
},
date_histogram => {
value_field => $field,
key_field => '@timestamp',
interval => "1m"
}
},
},
);
my @result;
for ( @{$data->{'facets'}->{'chart'}->{'entries'}} ) {
push @result, {
time => $_->{'time'},
count => $_->{'count'},
mean => sprintf "%.3f", $_->{'mean'} * 1000,
};
};
header('Content-Type' => 'application/json');
to_json(\@result); }; #主函数 package main; use File::Basename; #通过Tatsumaki::Application绑定urlpath到不同的类上。注意下面listhandler那里用的正则捕获。对,上面类里传参就是这么来的。注意最多不超过$9。 my $app = Tatsumaki::Application->new([
'/' => 'MainHandler',
'/api/chartdata' => 'ChartHandler',
'/api/(\w+)/(\w+)/(\w+)' => 'ListHandler', ]); #指定template和static的路径。类似Dancer里的views和public $app->template_path(dirname(__FILE__) . '/templates'); $app->static_path(dirname(__FILE__) . '/static'); #psgi app组建完成 return $app->psgi_app; true; ``` static里都是bootstrap的东西就不贴了。然后说说template。目前Tatsumaki只支持Text::MicroTemplate::File一种template,当然自己在handler里调用其他的template然后返回字符串也行。不过其实Text::MicroTemplate也蛮强大的。下面上例子: ```html %# 这就是Text::MicroTemplate强大的地方了,行首加个百分号就可以直接使用perl而不像TT那样尽量搞自己的语法 %# 配合render传来的handler(前面说了是类$self),整个环境全可以任意调用。 % my $mapping = $_[0]->{'mapping'}; % my $table = $_[0]->{'table'}; %# 比如这里其实就是通过handler调用request了。 % my $group = $_[0]->{handler}->args->[0]; % my @codes = qw(200 206 302 304 400 403 404 499 502 503 504); <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
</code></pre>
</div>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-type" content="text/html; charset=utf-8" />
<title>Bubble -- a perl webui for logstash & elasticsearch</title>
<link rel="stylesheet" href="/static/bootstrap/css/bootstrap.css" />
<link rel="stylesheet" href="/static/fontawesome/css/font-awesome.css" />
<link rel="stylesheet" href="/static/css/style.css" />
<link rel="stylesheet" href="/static/amcharts/style.css" type="text/css" />
</head>
<body>
<div class="container">
<div class="container-fluid">
<div class="row-fluid">
<div class="span3">
<div class="well sidebar-nav">
<ul class="nav nav-list">
<li class="nav-header">查询</li>
% for my $property ( @{ $mapping } ) {
% if ( ref( $property ) eq 'ARRAY' ) {
<li>@fields</li>
% for ( @{$property} ) {
<li> - <%= $_->{'name'} %> <%= $_->{'type'} %></li>
% }
% } elsif ( $property->{'type'} ) {
<li><%= $property->{'name'} %> <%= $property->{'type'} %></li>
% }
% }
</ul>
</div>
</div>
<div class="span9">
<div id="search">
<div class="control-group">
<div class="controls docs-input-sizes">
<select class="input-small inline" id="esapi" name="api">
<option>匹配方式</option>
<option>prefix</option>
<option>term</option>
<option>text</option>
</select>
<select class="input-small inline" id="eskey" name="key">
<option>查询列</option>
<option>oh</option>
<option>url</option>
</select>
<input type="text" class="input-big inline" id="esvalue" name="value" placeholder="查询文本" />
<input type="text" class="input-small inline" id="esstatus" name="status" placeholder="指定状态" />
<button class="btn btn-primary" onclick="genchart()">查看</button>
</div>
</div>
</div>
<div id="chartdiv" style="display:none; width: 100%; height: 500px;"></div>
<div id="estable">
% if ( $table ) {
<table class="table table-striped table-bordered table-condensed">
<thead>
<tr>
<th><%= $group %></th>
<th>平均响应时间</th>
<th>最大响应时间</th>
<th>下载数</th>
% for ( @codes ) {
<th><%= $_ %></th>
% }
</tr>
</thead>
<tbody>
% for my $list ( @{ $table } ) {
<tr>
<td><%= $list->{'key'} %></td>
<td><%= $list->{'mean'} %></td>
<td><%= $list->{'max'} %></td>
<td><%= $list->{'count'} %></td>
% for ( @codes ) {
% if ( $list->{'code'}->{$_} ) {
<td><%= $list->{'code'}->{$_} %></td>
% } else {
<td></td>
% }
% }
</tr>
% }
</tbody>
</table>
% }
</div>
</div>
</div>
</div>
</div>
<script src="/static/javascripts/jquery-1.7.2.min.js"></script>
<script src="/static/bootstrap/js/bootstrap.min.js"></script>
<script src="/static/amcharts/amstock.js" type="text/javascript"></script>
<script type="text/javascript">
var chart;
var chartProvider = [];
function createStockChart() {
chart = new AmCharts.AmStockChart();
chart.pathToImages = "/static/amcharts/images/";
var categoryAxesSettings = new AmCharts.CategoryAxesSettings();
categoryAxesSettings.parseDates = true;
categoryAxesSettings.minPeriod = "mm";
chart.categoryAxesSettings = categoryAxesSettings;
var dataSet = new AmCharts.DataSet();
dataSet.fieldMappings = [{
fromField : "count",
toField : "count",
}, {
fromField : "mean",
toField : "mean",
}];
dataSet.dataProvider = chartProvider;
dataSet.categoryField = "date";
chart.dataSets = [dataSet];
var stockPanel1 = new AmCharts.StockPanel();
stockPanel1.percentHeight = 70;
var valueAxis1 = new AmCharts.ValueAxis();
valueAxis1.position = "left";
valueAxis1.axisColor = "#999999";
stockPanel1.addValueAxis(valueAxis1);
var graph1 = new AmCharts.StockGraph();
graph1.valueField = "mean";
graph1.title = "mean(ms)";
graph1.type = "smoothedLine";
graph1.lineColor = "#999999";
graph1.fillAlphas = 0.2;
graph1.useDataSetColors = false;
stockPanel1.addStockGraph(graph1);
var stockLegend1 = new AmCharts.StockLegend();
stockPanel1.stockLegend = stockLegend1;
stockPanel1.drawingIconsEnabled = true;
var stockPanel2 = new AmCharts.StockPanel();
stockPanel2.percentHeight = 30;
stockPanel2.marginTop = 1;
stockPanel2.categoryAxis.dashLength = 5;
stockPanel2.showCategoryAxis = false;
valueAxis2 = new AmCharts.ValueAxis();
valueAxis2.dashLength = 5;
valueAxis2.gridAlpha = 0;
valueAxis2.axisThickness = 2;
stockPanel2.addValueAxis(valueAxis2);
var graph2 = new AmCharts.StockGraph();
graph2.valueAxis = valueAxis2;
graph2.valueField = "count";
graph2.title = "count";
graph2.balloonText = "[[value]]%";
graph2.type = "column";
graph2.cornerRadiusTop = 4;
graph2.fillAlphas = 1;
graph2.lineColor = "#FCD202";
graph2.useDataSetColors = false;
stockPanel2.addStockGraph(graph2);
var stockLegend2 = new AmCharts.StockLegend();
stockPanel2.stockLegend = stockLegend2;
chart.panels = [stockPanel1, stockPanel2];
var sbsettings = new AmCharts.ChartScrollbarSettings();
sbsettings.graph = graph2;
sbsettings.graphType = "line";
sbsettings.height = 30;
chart.chartScrollbarSettings = sbsettings;
var cursorSettings = new AmCharts.ChartCursorSettings();
cursorSettings.valueBalloonsEnabled = true;
chart.chartCursorSettings = cursorSettings;
$("#chartdiv").show();
chart.write("chartdiv");
};
function genchart() {
$.getJSON('/api/chartdata', {
api : $("#esapi").val(),
key : $("#eskey").val(),
status : $("#esstatus").val(),
value : $("#esvalue").val(),
}, function(data) {
for ( var i = 0; i < data.length; i++ ) {
var date = new Date(data[i].time);
chartProvider.push({
date: date,
count: data[i].count,
mean: data[i].mean,
});
}
createStockChart();
});
};
</script>
</body>
</html>
<p>```</p>
<p>效果如下:</p>
<p><img src="/images/uploads/tatsumaki.png" alt="查询表格并提交最多次的url绘图" title="查询表格并提交最多次的url绘图" /></p>
<hr />
<p><strong>2012 年 12 月 30 日附注:</strong></p>
<p>更好的纯 js 版本已经作为独立的 elasticsearch-plugin 项目发布在 github 上。地址:<a href="https://github.com/chenryn/elasticsearch-logstash-faceter">https://github.com/chenryn/elasticsearch-logstash-faceter</a> 。欢迎大家试用!!</p>
用gnuplot绘制多图
2012-11-22T00:00:00+08:00
monitor
gnuplot
http://chenlinux.com/2012/11/22/gnuplot-to-draw-multi-graph
<p>以前已经提过多次gnuplot的简便快捷了。不过大多是最基本的单图上画条线之类的。这次碰到需求,稍微help了一下在一个图上画多个区域。主要需要注意的就是set size的定位点到底从什么角度算,说实话蛮麻烦的。</p>
<p>上文件:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">result</span><span class="o">=</span><span class="nv">$1</span>
<span class="nv">begin</span><span class="o">=</span><span class="sb">`</span>head -n 1 <span class="nv">$result</span>.txt | awk <span class="s1">'{print $1}'</span><span class="sb">`</span>
<span class="nv">end</span><span class="o">=</span><span class="sb">`</span>tail -n 1 <span class="nv">$result</span>.txt | awk <span class="s1">'{print $1}'</span><span class="sb">`</span>
cat > conf/<span class="nv">$result</span>.conf <span class="sh"><<EOF
set terminal png
set output "png/$result.png"
set multiplot
set xdata time
set timefmt "%H:%M:%S"
set format x "%M:%S"
set size 1.0,0.5
set origin 0.0,0.5
set ylabel "KRps"
unset xtics
plot "res/$result.txt" using 1:(\$2/1000) with points linewidth 2 title ""
set origin 0.0,0.0
set size 1.0,0.35
set xtics
set xrange ["$begin":"$end"]
set ylabel "%usr:sys:irq"
plot "res/$result.csv" using 2:(\$3+\$4+\$8) with boxes fs solid 1.0 title "", \
"res/$result.csv" using 2:(\$3+\$4) with boxes fs solid 1.0 title "", \
"res/$result.csv" using 2:3 with boxes fs solid 1.0 title ""
set origin 0.0,0.3
set size 1.0,0.25
unset xtics
set ylabel "MBps"
plot "res/$result.csv" using 2:(\$11/1024/1024) with boxes fs solid 0.7 linecolor rgb "green" title "", \
"res/$result.csv" using 2:(\$12/1024/1024) with lines linewidth 2 linecolor rgb "blue" title ""
EOF
</span> cat conf/<span class="nv">$result</span>.conf | gnuplot
</code></pre>
</div>
<p>注意:新增了一行xrange配置,如果不指定这个几张小图的xtics会不统一,而上面两张图的xtics又已经被unset了,结果看起来就跟不同步似的。</p>
<p>效果如下:<br />
<img src="/images/uploads/gnuplot-multi.png" alt="图片" /></p>
syslog实时报警"说出来"
2012-11-19T00:00:00+08:00
monitor
websocket
perl
syslog
http://chenlinux.com/2012/11/19/syslog-realtime-warning-to-speech
<p>syslog应该是大家最常用的,也基本可以说是最重要的服务器监控信息来源了。</p>
<p>syslog的传输,应该不用再说,哪怕在百度里搜都有足够多的靠谱结果。而关于报警的问题,之前我也写了好几篇,比如<a href="/2012/10/17/juggernaut-for-syslog-check">《用Juggernaut实时推送syslog分析结果》</a>讲了如何用websocket推送结果,<a href="/2012/11/09/chrome-app-demo">《Chrome的APP简单用法》</a>讲了如何利用chrome后台页面开机自动运行进行桌面提示。</p>
<p>那么,如果我既不想开网页看,也不好安装chrome浏览器,有没有够简便的办法接收呢?有!Linux社区从来不缺乏各种神奇工具。下面介绍两个同样强大的提示办法。</p>
<p>第一个,非chrome型的桌面通知notify-send命令,依发行版不同,可能属于libnotify-tools或者libnotify-bin包,自己搜索即可;</p>
<p>第二个,Espeak命令,著名Text To Speech软件,虽然电子音怪了点,但是支持中文而且文件很小,同样直接在源里安装即可。</p>
<p>下面就是如何把这两个强大的工具和server结合起来的问题了,出动胶水语言代表perl。代码如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nn">Mojo::</span><span class="nv">UserAgent</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">JSON</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$ua</span> <span class="o">=</span> <span class="nn">Mojo::</span><span class="nv">UserAgent</span><span class="o">-></span><span class="k">new</span><span class="p">();</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$sid</span><span class="p">,</span> <span class="nv">$ws</span> <span class="p">);</span>
<span class="c1"># 本来用Protocol::WebSocket::Handshake::Client模块,指定IP和端口,自动会获取sid拼ws地址的,不过测试发现open后没反应。奇怪</span>
<span class="nv">LABEL:</span>
<span class="nv">$sid</span> <span class="o">=</span> <span class="p">(</span><span class="o">+</span><span class="nb">split</span><span class="p">(</span><span class="sr">/:/</span><span class="p">,</span> <span class="nv">$ua</span><span class="o">-></span><span class="nv">get</span><span class="p">(</span><span class="s">'http://syslog.domain.com:8080/socket.io/1/'</span><span class="p">)</span><span class="o">-></span><span class="nv">res</span><span class="o">-></span><span class="nv">body</span><span class="p">))[</span><span class="mi">0</span><span class="p">];</span>
<span class="nv">$ws</span> <span class="o">=</span> <span class="s">"ws://syslog.domain.com:8080/socket.io/1/websocket/${sid}"</span><span class="p">;</span>
<span class="nv">$ua</span><span class="o">-></span><span class="nv">websocket</span><span class="p">(</span> <span class="nv">$ws</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$ua</span><span class="p">,</span> <span class="nv">$tx</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="nv">$tx</span><span class="o">-></span><span class="nb">send</span><span class="p">(</span><span class="s">'3:::{"type":"subscribe","channel":"syslog"}'</span><span class="p">);</span>
<span class="nv">$tx</span><span class="o">-></span><span class="nv">on</span><span class="p">(</span><span class="nv">finish</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="c1"># 很怪的是,mojo::useragent的websocket client总是在不到一分钟内就进入on_finish状态,所以这里只好返回重连</span>
<span class="nn">Mojo::</span><span class="nv">IOLoop</span><span class="o">-></span><span class="nv">stop</span><span class="p">;</span>
<span class="nb">goto</span> <span class="nv">LABEL</span><span class="p">;</span>
<span class="p">});</span>
<span class="nv">$tx</span><span class="o">-></span><span class="nv">on</span><span class="p">(</span><span class="nv">message</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$tx</span><span class="p">,</span> <span class="nv">$msg</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span> <span class="nb">length</span><span class="p">(</span> <span class="nv">$msg</span> <span class="p">)</span> <span class="o">></span> <span class="mi">5</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$syslog</span> <span class="o">=</span> <span class="nv">from_json</span><span class="p">(</span> <span class="nb">substr</span><span class="p">(</span> <span class="nv">$msg</span><span class="p">,</span> <span class="mi">3</span> <span class="p">)</span> <span class="p">);</span>
<span class="nv">notify</span><span class="p">(</span> <span class="nv">$syslog</span> <span class="p">);</span>
<span class="p">};</span>
<span class="p">});</span>
<span class="p">});</span>
<span class="nn">Mojo::</span><span class="nv">IOLoop</span><span class="o">-></span><span class="nv">start</span> <span class="k">unless</span> <span class="nn">Mojo::</span><span class="nv">IOLoop</span><span class="o">-></span><span class="nv">is_running</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">notify</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$data</span> <span class="o">=</span> <span class="nv">$_</span><span class="p">;</span>
<span class="k">return</span> <span class="k">if</span> <span class="nv">$data</span><span class="o">-></span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="ow">eq</span> <span class="s">'btn'</span><span class="p">;</span>
<span class="nb">exec</span><span class="p">(</span><span class="s">"notify-send \"$data->[0] $data->[1]\" \"$data->[3] $data->[4]\""</span><span class="p">);</span>
<span class="c1"># 注意设定-s 120,默认是175,念得飞快</span>
<span class="nb">exec</span><span class="p">(</span><span class="s">"espeak -vzh+f2 -s 120 \"$data->[1]\""</span><span class="p">);</span> <span class="c1"># 指定中文报ip,不然很难听懂</span>
<span class="c1"># f是女生,m是男声,至于第几个声音,我没听出来多大差别,都跟九十年代初电影里的机器人一样</span>
<span class="nb">exec</span><span class="p">(</span><span class="s">"espeak -ven+m2 -s 120 \"$data->[3] $data->[4]\""</span><span class="p">);</span> <span class="c1"># 指定英文报内容,不然用中文的声音念更难听懂</span>
<span class="p">};</span>
</code></pre>
</div>
<p>以上抛砖引玉,大家可以试试Ekho(余音),这是国人开发的真人语音TTS开源软件,还支持粤语,文言文等选择,汗……</p>
【翻译】用ElasticSearch和Protovis实现数据可视化
2012-11-18T00:00:00+08:00
logstash
elasticsearch
javascript
http://chenlinux.com/2012/11/18/data-visualization-with-elasticsearch-and-protovis
<p>搜索引擎最重要的目的,嗯,不出意料就是<code class="highlighter-rouge">搜索</code>。你传给它一个请求,然后它依照相关性返回你一串匹配的结果。我们可以根据自己的内容创造各种请求结构,试验各种不同的分析器,搜索引擎都会努力尝试提供最好的结果。</p>
<p>不过,一个现代的全文搜索引擎可以做的比这个更多。因为它的核心是基于一个为了高效查询匹配文档而高度优化过的数据结构——<a href="http://en.wikipedia.org/wiki/Index_\(search_engine\)#Inverted_indices">倒排索引</a>。它也可以为我们的数据完成复杂的<code class="highlighter-rouge">聚合</code>运算,在这里我们叫它facets。(不好翻译,后文对这个单词都保留英文)</p>
<p>facets通常的目的是提供给用户某个方面的导航或者搜索。 当你在网上商店搜索“相机”,你可以选择不同的制造商,价格范围或者特定功能来定制条件,这应该就是点一下链接的事情,而不是通过修改一长串查询语法。</p>
<p>一个<a href="http://blog.linkedin.com/2009/12/14/linkedin-faceted-search/">LinkedIn的导航</a>范例如下图所示:</p>
<p><img src="http://www.elasticsearch.cn/blog/images/dashboards/linkedin-faceted-search.png" alt="图片1" /></p>
<p>Facet搜索为数不多的几个可以把强大的请求能力开放给最终用户的办法之一,详见Moritz Stefaner的试验<a href="http://well-formed-data.net/archives/54/elastic-lists">“Elastic Lists”</a>,或许你会有更多灵感。</p>
<p>但是,除了链接和复选框,其实我们还能做的更多。比如利用这些数据画图,而这就是我们在这篇文章中要讲的。</p>
<h1 id="section">实时仪表板</h1>
<p>在几乎所有的分析、监控和数据挖掘服务中,或早或晚的你都会碰到这样的需求:“我们要一个仪表板!”。因为大家都爱仪表板,可能因为真的有用,可能单纯因为它漂亮~这时候,我们不用写任何<a href="http://en.wikipedia.org/wiki/Online_analytical_processing">OLAP</a>实现,用facets就可以完成一个很漂亮很给力的分析引擎。</p>
<p>下面的截图就是从一个<a href="http://ataxosocialinsider.cz/">社交媒体监控应用</a>上获取的。这个应用不单用ES来搜索和挖掘数据,还通过交互式仪表板提供数据聚合功能。</p>
<p><img src="http://www.elasticsearch.cn/blog/images/dashboards/dashboard.png" alt="图片2" /></p>
<p>当用户深入数据,添加一个关键字,使用一个自定义查询,所有的图都会实时更新,这就是facet聚合的工作方式。仪表板上不是数据定期计算好的的静态快照,而是一个用于数据探索的真正的交互式工具。</p>
<p>在本文中,我们将会学习到怎样从ES中获取数据,然后怎么创建这些图表。</p>
<h1 id="terms-facet">关系聚合(terms facet)的饼图</h1>
<p>第一个图,我们用ES中比较简单的<a href="http://elasticsearch.org/guide/reference/api/search/facets/terms-facet.html">terms</a>facet来做。这个facet会返回一个字段中最常见的词汇和它的计数值。</p>
<p>首先我们先插入一些数据。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>curl -X DELETE <span class="s2">"http://localhost:9200/dashboard"</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'
{ "title" : "One",
"tags" : ["ruby", "java", "search"]}
'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'
{ "title" : "Two",
"tags" : ["java", "search"] }
'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'
{ "title" : "Three",
"tags" : ["erlang", "search"] }
'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'
{ "title" : "Four",
"tags" : ["search"] }
'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/_refresh"</span>
</code></pre>
</div>
<p>你们都看到了,我们存储了一些文章的标签,每个文章可以多个标签,数据以JSON格式发送,这也是ES的文档格式。</p>
<p>现在,要知道文档的十大标签,我们只需要简单的请求:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>curl -X POST <span class="s2">"http://localhost:9200/dashboard/_search?pretty=true"</span> -d <span class="s1">'
{
"query" : { "match_all" : {} },
"facets" : {
"tags" : { "terms" : {"field" : "tags", "size" : 10} }
}
}
'</span>
</code></pre>
</div>
<p>你看到了,我接受所有文档,然后定义一个terms facet叫做“tags”。这个请求会返回如下样子的数据:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span>
<span class="s2">"took"</span> <span class="err">:</span> <span class="mi">2</span><span class="p">,</span>
<span class="c1">// ... snip ...</span>
<span class="s2">"hits"</span> <span class="err">:</span> <span class="p">{</span>
<span class="s2">"total"</span> <span class="err">:</span> <span class="mi">4</span><span class="p">,</span>
<span class="c1">// ... snip ...</span>
<span class="p">},</span>
<span class="s2">"facets"</span> <span class="err">:</span> <span class="p">{</span>
<span class="s2">"tags"</span> <span class="err">:</span> <span class="p">{</span>
<span class="s2">"_type"</span> <span class="err">:</span> <span class="s2">"terms"</span><span class="p">,</span>
<span class="s2">"missing"</span> <span class="err">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s2">"terms"</span> <span class="err">:</span> <span class="p">[</span>
<span class="p">{</span> <span class="s2">"term"</span> <span class="p">:</span> <span class="s2">"search"</span><span class="p">,</span> <span class="s2">"count"</span> <span class="p">:</span> <span class="mi">4</span> <span class="p">},</span>
<span class="p">{</span> <span class="s2">"term"</span> <span class="p">:</span> <span class="s2">"java"</span><span class="p">,</span> <span class="s2">"count"</span> <span class="p">:</span> <span class="mi">2</span> <span class="p">},</span>
<span class="p">{</span> <span class="s2">"term"</span> <span class="p">:</span> <span class="s2">"ruby"</span><span class="p">,</span> <span class="s2">"count"</span> <span class="p">:</span> <span class="mi">1</span> <span class="p">},</span>
<span class="p">{</span> <span class="s2">"term"</span> <span class="p">:</span> <span class="s2">"erlang"</span><span class="p">,</span> <span class="s2">"count"</span> <span class="p">:</span> <span class="mi">1</span> <span class="p">}</span>
<span class="p">]</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>JSON中<code class="highlighter-rouge">facets</code>部分是我们关心的,特别是<code class="highlighter-rouge">facets.tags.terms</code>数组。它告诉我们有四篇文章打了search标签,两篇java标签,等等…….(当然,我们或许应该给请求添加一个<code class="highlighter-rouge">size</code>参数跳过前面的结果)</p>
<p>这种比例类型的数据最合适的可视化方案就是饼图,或者它的变体:油炸圈饼图。最终结果如下(你可能希望看这个<a href="http://www.elasticsearch.cn/blog/assets/dashboards/donut.html">可运行的实例</a>):</p>
<p><img src="http://www.elasticsearch.cn/blog/images/dashboards/donut_chart.png" alt="图片3" /></p>
<p>我们将使用<a href="http://vis.stanford.edu/protovis/">Protovis</a>一个JavaScript的数据可视化工具集。Protovis是100%开源的,你可以想象它是数据可视化方面的RoR。和其他类似工具形成鲜明对比的是,它没有附带一组图标类型来供你“选择”。而是定义了一组原语和一个灵活的DSL,这样你可以非常简单的创建自定义的可视化。创建<a href="http://vis.stanford.edu/protovis/ex/pie.html">饼图</a>就非常简单。</p>
<p>因为ES返回的是JSON数据,我们可以通过Ajax调用加载它。不要忘记你可以clone或者下载实例的<a href="https://gist.github.com/966338">全部源代码</a>。</p>
<p>首先需要一个HTML文件来容纳图标然后从ES里加载数据:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="cp"><!DOCTYPE html></span>
<span class="nt"><html></span>
<span class="nt"><head></span>
<span class="nt"><title></span>ElasticSearch Terms Facet Donut Chart<span class="nt"></title></span>
<span class="nt"><meta</span> <span class="na">http-equiv=</span><span class="s">"Content-Type"</span> <span class="na">content=</span><span class="s">"text/html; charset=utf-8"</span> <span class="nt">/></span>
<span class="c"><!-- Load JS libraries --></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"jquery-1.5.1.min.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"protovis-r3.2.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"donut.js"</span><span class="nt">></script></span>
<span class="nt"><script></span>
<span class="nx">$</span><span class="p">(</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span> <span class="nx">load_data</span><span class="p">();</span> <span class="p">});</span>
<span class="kd">var</span> <span class="nx">load_data</span> <span class="o">=</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">.</span><span class="nx">ajax</span><span class="p">({</span> <span class="na">url</span><span class="p">:</span> <span class="s1">'http://localhost:9200/dashboard/article/_search?pretty=true'</span>
<span class="p">,</span> <span class="na">type</span><span class="p">:</span> <span class="s1">'POST'</span>
<span class="p">,</span> <span class="na">data</span> <span class="p">:</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">stringify</span><span class="p">({</span>
<span class="s2">"query"</span> <span class="p">:</span> <span class="p">{</span> <span class="s2">"match_all"</span> <span class="p">:</span> <span class="p">{}</span> <span class="p">},</span>
<span class="s2">"facets"</span> <span class="p">:</span> <span class="p">{</span>
<span class="s2">"tags"</span> <span class="p">:</span> <span class="p">{</span>
<span class="s2">"terms"</span> <span class="p">:</span> <span class="p">{</span>
<span class="s2">"field"</span> <span class="p">:</span> <span class="s2">"tags"</span><span class="p">,</span>
<span class="s2">"size"</span> <span class="p">:</span> <span class="s2">"10"</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">})</span>
<span class="p">,</span> <span class="na">dataType</span> <span class="p">:</span> <span class="s1">'json'</span>
<span class="p">,</span> <span class="na">processData</span><span class="p">:</span> <span class="kc">false</span>
<span class="p">,</span> <span class="na">success</span><span class="p">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">json</span><span class="p">,</span> <span class="nx">statusText</span><span class="p">,</span> <span class="nx">xhr</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">display_chart</span><span class="p">(</span><span class="nx">json</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">,</span> <span class="na">error</span><span class="p">:</span> <span class="kd">function</span><span class="p">(</span><span class="nx">xhr</span><span class="p">,</span> <span class="nx">message</span><span class="p">,</span> <span class="nx">error</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="s2">"Error while loading data from ElasticSearch"</span><span class="p">,</span> <span class="nx">message</span><span class="p">);</span>
<span class="k">throw</span><span class="p">(</span><span class="nx">error</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="kd">var</span> <span class="nx">display_chart</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">json</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">Donut</span><span class="p">().</span><span class="nx">data</span><span class="p">(</span><span class="nx">json</span><span class="p">.</span><span class="nx">facets</span><span class="p">.</span><span class="nx">tags</span><span class="p">.</span><span class="nx">terms</span><span class="p">).</span><span class="nx">draw</span><span class="p">();</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="nt"></script></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="c"><!-- Placeholder for the chart --></span>
<span class="nt"><div</span> <span class="na">id=</span><span class="s">"chart"</span><span class="nt">></div></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</code></pre>
</div>
<p>文档加载后,我们通过Ajax收到和之前<code class="highlighter-rouge">curl</code>测试中一样的facet。在jQuery的Ajaxcallback里我们通过封装的<code class="highlighter-rouge">display_chart()</code>把返回的JSON传给<code class="highlighter-rouge">Donut()</code>函数.</p>
<p><code class="highlighter-rouge">Donut()</code>函数及注释如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">// =====================================================================================================</span>
<span class="c1">// A donut chart with Protovis - See http://vis.stanford.edu/protovis/ex/pie.html</span>
<span class="c1">// =====================================================================================================</span>
<span class="kd">var</span> <span class="nx">Donut</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">dom_id</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="s1">'undefined'</span> <span class="o">==</span> <span class="k">typeof</span> <span class="nx">dom_id</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// Set the default DOM element ID to bind</span>
<span class="nx">dom_id</span> <span class="o">=</span> <span class="s1">'chart'</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">var</span> <span class="nx">data</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">json</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// Set the data for the chart</span>
<span class="k">this</span><span class="p">.</span><span class="nx">data</span> <span class="o">=</span> <span class="nx">json</span><span class="p">;</span>
<span class="k">return</span> <span class="k">this</span><span class="p">;</span>
<span class="p">};</span>
<span class="kd">var</span> <span class="nx">draw</span> <span class="o">=</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">entries</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">data</span><span class="p">.</span><span class="nx">sort</span><span class="p">(</span> <span class="kd">function</span><span class="p">(</span><span class="nx">a</span><span class="p">,</span> <span class="nx">b</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// Sort the data by term names, so the</span>
<span class="k">return</span> <span class="nx">a</span><span class="p">.</span><span class="nx">term</span> <span class="o"><</span> <span class="nx">b</span><span class="p">.</span><span class="nx">term</span> <span class="p">?</span> <span class="o">-</span><span class="mi">1</span> <span class="p">:</span> <span class="mi">1</span><span class="p">;</span> <span class="c1">// color scheme for wedges is preserved</span>
<span class="p">}),</span> <span class="c1">// with any order</span>
<span class="nx">values</span> <span class="o">=</span> <span class="nx">pv</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">entries</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// Create an array holding just the counts</span>
<span class="k">return</span> <span class="nx">e</span><span class="p">.</span><span class="nx">count</span><span class="p">;</span>
<span class="p">});</span>
<span class="c1">// console.log('Drawing', entries, values);</span>
<span class="kd">var</span> <span class="nx">w</span> <span class="o">=</span> <span class="mi">200</span><span class="p">,</span> <span class="c1">// Dimensions and color scheme for the chart</span>
<span class="nx">h</span> <span class="o">=</span> <span class="mi">200</span><span class="p">,</span>
<span class="nx">colors</span> <span class="o">=</span> <span class="nx">pv</span><span class="p">.</span><span class="nx">Colors</span><span class="p">.</span><span class="nx">category10</span><span class="p">().</span><span class="nx">range</span><span class="p">();</span>
<span class="kd">var</span> <span class="nx">vis</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">pv</span><span class="p">.</span><span class="nx">Panel</span><span class="p">()</span> <span class="c1">// Create the basis panel</span>
<span class="p">.</span><span class="nx">width</span><span class="p">(</span><span class="nx">w</span><span class="p">)</span>
<span class="p">.</span><span class="nx">height</span><span class="p">(</span><span class="nx">h</span><span class="p">)</span>
<span class="p">.</span><span class="nx">margin</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
<span class="nx">vis</span><span class="p">.</span><span class="nx">add</span><span class="p">(</span><span class="nx">pv</span><span class="p">.</span><span class="nx">Wedge</span><span class="p">)</span> <span class="c1">// Create the "wedges" of the chart</span>
<span class="p">.</span><span class="nx">def</span><span class="p">(</span><span class="s2">"active"</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="c1">// Auxiliary variable to hold mouse over state</span>
<span class="p">.</span><span class="nx">data</span><span class="p">(</span> <span class="nx">pv</span><span class="p">.</span><span class="nx">normalize</span><span class="p">(</span><span class="nx">values</span><span class="p">)</span> <span class="p">)</span> <span class="c1">// Pass the normalized data to Protovis</span>
<span class="p">.</span><span class="nx">left</span><span class="p">(</span><span class="nx">w</span><span class="o">/</span><span class="mi">3</span><span class="p">)</span> <span class="c1">// Set-up chart position and dimension</span>
<span class="p">.</span><span class="nx">top</span><span class="p">(</span><span class="nx">w</span><span class="o">/</span><span class="mi">3</span><span class="p">)</span>
<span class="p">.</span><span class="nx">outerRadius</span><span class="p">(</span><span class="nx">w</span><span class="o">/</span><span class="mi">3</span><span class="p">)</span>
<span class="p">.</span><span class="nx">innerRadius</span><span class="p">(</span><span class="mi">15</span><span class="p">)</span> <span class="c1">// Create a "donut hole" in the center</span>
<span class="p">.</span><span class="nx">angle</span><span class="p">(</span> <span class="kd">function</span><span class="p">(</span><span class="nx">d</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// Compute the "width" of the wedge</span>
<span class="k">return</span> <span class="nx">d</span> <span class="o">*</span> <span class="mi">2</span> <span class="o">*</span> <span class="nb">Math</span><span class="p">.</span><span class="nx">PI</span><span class="p">;</span>
<span class="p">})</span>
<span class="p">.</span><span class="nx">strokeStyle</span><span class="p">(</span><span class="s2">"#fff"</span><span class="p">)</span> <span class="c1">// Add white stroke</span>
<span class="p">.</span><span class="nx">event</span><span class="p">(</span><span class="s2">"mouseover"</span><span class="p">,</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span> <span class="c1">// On "mouse over", set the "wedge" as active</span>
<span class="k">this</span><span class="p">.</span><span class="nx">active</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">index</span><span class="p">);</span>
<span class="k">this</span><span class="p">.</span><span class="nx">cursor</span><span class="p">(</span><span class="s1">'pointer'</span><span class="p">);</span>
<span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">root</span><span class="p">.</span><span class="nx">render</span><span class="p">();</span>
<span class="p">})</span>
<span class="p">.</span><span class="nx">event</span><span class="p">(</span><span class="s2">"mouseout"</span><span class="p">,</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span> <span class="c1">// On "mouse out", clear the active state</span>
<span class="k">this</span><span class="p">.</span><span class="nx">active</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">);</span>
<span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">root</span><span class="p">.</span><span class="nx">render</span><span class="p">();</span>
<span class="p">})</span>
<span class="p">.</span><span class="nx">event</span><span class="p">(</span><span class="s2">"mousedown"</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">d</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// On "mouse down", perform action,</span>
<span class="kd">var</span> <span class="nx">term</span> <span class="o">=</span> <span class="nx">entries</span><span class="p">[</span><span class="k">this</span><span class="p">.</span><span class="nx">index</span><span class="p">].</span><span class="nx">term</span><span class="p">;</span> <span class="c1">// such as filtering the results...</span>
<span class="k">return</span> <span class="p">(</span><span class="nx">alert</span><span class="p">(</span><span class="s2">"Filter the results by '"</span><span class="o">+</span><span class="nx">term</span><span class="o">+</span><span class="s2">"'"</span><span class="p">));</span>
<span class="p">})</span>
<span class="p">.</span><span class="nx">anchor</span><span class="p">(</span><span class="s2">"right"</span><span class="p">).</span><span class="nx">add</span><span class="p">(</span><span class="nx">pv</span><span class="p">.</span><span class="nx">Dot</span><span class="p">)</span> <span class="c1">// Add the left part of he "inline" label,</span>
<span class="c1">// displayed inside the donut "hole"</span>
<span class="p">.</span><span class="nx">visible</span><span class="p">(</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span> <span class="c1">// The label is visible when its wedge is active</span>
<span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">parent</span><span class="p">.</span><span class="nx">children</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="p">.</span><span class="nx">active</span><span class="p">()</span> <span class="o">==</span> <span class="k">this</span><span class="p">.</span><span class="nx">index</span><span class="p">;</span>
<span class="p">})</span>
<span class="p">.</span><span class="nx">fillStyle</span><span class="p">(</span><span class="s2">"#222"</span><span class="p">)</span>
<span class="p">.</span><span class="nx">lineWidth</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="p">.</span><span class="nx">radius</span><span class="p">(</span><span class="mi">14</span><span class="p">)</span>
<span class="p">.</span><span class="nx">anchor</span><span class="p">(</span><span class="s2">"center"</span><span class="p">).</span><span class="nx">add</span><span class="p">(</span><span class="nx">pv</span><span class="p">.</span><span class="nx">Bar</span><span class="p">)</span> <span class="c1">// Add the middle part of the label</span>
<span class="p">.</span><span class="nx">fillStyle</span><span class="p">(</span><span class="s2">"#222"</span><span class="p">)</span>
<span class="p">.</span><span class="nx">width</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">d</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// Compute width:</span>
<span class="k">return</span> <span class="p">(</span><span class="nx">d</span><span class="o">*</span><span class="mi">100</span><span class="p">).</span><span class="nx">toFixed</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="c1">// add pixels for percents</span>
<span class="p">.</span><span class="nx">toString</span><span class="p">().</span><span class="nx">length</span><span class="o">*</span><span class="mi">4</span> <span class="o">+</span>
<span class="mi">10</span> <span class="o">+</span> <span class="c1">// add pixels for glyphs (%, etc)</span>
<span class="nx">entries</span><span class="p">[</span><span class="k">this</span><span class="p">.</span><span class="nx">index</span><span class="p">]</span> <span class="c1">// add pixels for letters (very rough)</span>
<span class="p">.</span><span class="nx">term</span><span class="p">.</span><span class="nx">length</span><span class="o">*</span><span class="mi">9</span><span class="p">;</span>
<span class="p">})</span>
<span class="p">.</span><span class="nx">height</span><span class="p">(</span><span class="mi">28</span><span class="p">)</span>
<span class="p">.</span><span class="nx">top</span><span class="p">((</span><span class="nx">w</span><span class="o">/</span><span class="mi">3</span><span class="p">)</span><span class="o">-</span><span class="mi">14</span><span class="p">)</span>
<span class="p">.</span><span class="nx">anchor</span><span class="p">(</span><span class="s2">"right"</span><span class="p">).</span><span class="nx">add</span><span class="p">(</span><span class="nx">pv</span><span class="p">.</span><span class="nx">Dot</span><span class="p">)</span> <span class="c1">// Add the right part of the label</span>
<span class="p">.</span><span class="nx">fillStyle</span><span class="p">(</span><span class="s2">"#222"</span><span class="p">)</span>
<span class="p">.</span><span class="nx">lineWidth</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="p">.</span><span class="nx">radius</span><span class="p">(</span><span class="mi">14</span><span class="p">)</span>
<span class="p">.</span><span class="nx">parent</span><span class="p">.</span><span class="nx">children</span><span class="p">[</span><span class="mi">2</span><span class="p">].</span><span class="nx">anchor</span><span class="p">(</span><span class="s2">"left"</span><span class="p">)</span> <span class="c1">// Add the text to label</span>
<span class="p">.</span><span class="nx">add</span><span class="p">(</span><span class="nx">pv</span><span class="p">.</span><span class="nx">Label</span><span class="p">)</span>
<span class="p">.</span><span class="nx">left</span><span class="p">((</span><span class="nx">w</span><span class="o">/</span><span class="mi">3</span><span class="p">)</span><span class="o">-</span><span class="mi">7</span><span class="p">)</span>
<span class="p">.</span><span class="nx">text</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">d</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// Combine the text for label</span>
<span class="k">return</span> <span class="p">(</span><span class="nx">d</span><span class="o">*</span><span class="mi">100</span><span class="p">).</span><span class="nx">toFixed</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">+</span> <span class="s2">"%"</span> <span class="o">+</span>
<span class="s1">' '</span> <span class="o">+</span> <span class="nx">entries</span><span class="p">[</span><span class="k">this</span><span class="p">.</span><span class="nx">index</span><span class="p">].</span><span class="nx">term</span> <span class="o">+</span>
<span class="s1">' ('</span> <span class="o">+</span> <span class="nx">values</span><span class="p">[</span><span class="k">this</span><span class="p">.</span><span class="nx">index</span><span class="p">]</span> <span class="o">+</span> <span class="s1">')'</span><span class="p">;</span>
<span class="p">})</span>
<span class="p">.</span><span class="nx">textStyle</span><span class="p">(</span><span class="s2">"#fff"</span><span class="p">)</span>
<span class="p">.</span><span class="nx">root</span><span class="p">.</span><span class="nx">canvas</span><span class="p">(</span><span class="nx">dom_id</span><span class="p">)</span> <span class="c1">// Bind the chart to DOM element</span>
<span class="p">.</span><span class="nx">render</span><span class="p">();</span> <span class="c1">// And render it.</span>
<span class="p">};</span>
<span class="k">return</span> <span class="p">{</span> <span class="c1">// Create the public API</span>
<span class="na">data</span> <span class="p">:</span> <span class="nx">data</span><span class="p">,</span>
<span class="na">draw</span> <span class="p">:</span> <span class="nx">draw</span>
<span class="p">};</span>
<span class="p">};</span>
</code></pre>
</div>
<p>现在你们看到了,一个简单的JSON数据转换,我们就可以创建出丰富的有吸引力的关于我们文章标签分布的可视化图标。完整的例子在<a href="http://www.elasticsearch.cn/blog/assets/dashboards/donut.html">这里</a>。</p>
<p>当你使用完全不同的请求,比如显示某个特定作者的文章,或者特定日期内发表的文章,整个可视化都照样正常工作,代码是可以重用的。</p>
<h1 id="date-histogram-facets">日期直方图(date histogram facets)时间线</h1>
<p>Protovis让创建另一种常见的可视化类型也非常容易:<a href="http://vis.stanford.edu/protovis/ex/zoom.html">时间线</a>。任何类型的数据,只要和特定日期相关的,比如文章发表,事件发生,目标达成,都可以被可视化成时间线。</p>
<p>最终结果就像下面这样(同样可以看<a href="http://www.elasticsearch.cn/blog/assets/dashboards/timeline.html">运行版</a>):</p>
<p><img src="http://www.elasticsearch.cn/blog/images/dashboards/timeline_chart.png" alt="图片4" /></p>
<p>好了,让我们往索引里存一些带有发表日期的文章吧:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>curl -X DELETE <span class="s2">"http://localhost:9200/dashboard"</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "1", "published" : "2011-01-01" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "2", "published" : "2011-01-02" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "3", "published" : "2011-01-02" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "4", "published" : "2011-01-03" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "5", "published" : "2011-01-04" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "6", "published" : "2011-01-04" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "7", "published" : "2011-01-04" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "8", "published" : "2011-01-04" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "9", "published" : "2011-01-10" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "10", "published" : "2011-01-12" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "11", "published" : "2011-01-13" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "12", "published" : "2011-01-14" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "13", "published" : "2011-01-14" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "14", "published" : "2011-01-15" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "15", "published" : "2011-01-20" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "16", "published" : "2011-01-20" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "17", "published" : "2011-01-21" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "18", "published" : "2011-01-22" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "19", "published" : "2011-01-23" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/article"</span> -d <span class="s1">'{ "t" : "20", "published" : "2011-01-24" }'</span>
curl -X POST <span class="s2">"http://localhost:9200/dashboard/_refresh"</span>
</code></pre>
</div>
<p>我们用ES的<a href="http://www.elasticsearch.org/guide/reference/api/search/facets/date-histogram-facet.html">date histogram facet</a>来获取文章发表的频率。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>curl -X POST <span class="s2">"http://localhost:9200/dashboard/_search?pretty=true"</span> -d <span class="s1">'
{
"query" : { "match_all" : {} },
"facets" : {
"published_on" : {
"date_histogram" : {
"field" : "published",
"interval" : "day"
}
}
}
}
'</span>
</code></pre>
</div>
<p>注意我们是怎么设置间隔为天的。这个很容易就可以替换成周,月 ,或者年。</p>
<p>请求会返回像下面这样的JSON:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span>
<span class="s2">"took"</span> <span class="err">:</span> <span class="mi">2</span><span class="p">,</span>
<span class="c1">// ... snip ...</span>
<span class="s2">"hits"</span> <span class="err">:</span> <span class="p">{</span>
<span class="s2">"total"</span> <span class="err">:</span> <span class="mi">4</span><span class="p">,</span>
<span class="c1">// ... snip ...</span>
<span class="p">},</span>
<span class="s2">"facets"</span> <span class="err">:</span> <span class="p">{</span>
<span class="s2">"published"</span> <span class="err">:</span> <span class="p">{</span>
<span class="s2">"_type"</span> <span class="err">:</span> <span class="s2">"histogram"</span><span class="p">,</span>
<span class="s2">"entries"</span> <span class="err">:</span> <span class="p">[</span>
<span class="p">{</span> <span class="s2">"time"</span> <span class="p">:</span> <span class="mi">1293840000000</span><span class="p">,</span> <span class="s2">"count"</span> <span class="p">:</span> <span class="mi">1</span> <span class="p">},</span>
<span class="p">{</span> <span class="s2">"time"</span> <span class="p">:</span> <span class="mi">1293926400000</span><span class="p">,</span> <span class="s2">"count"</span> <span class="p">:</span> <span class="mi">2</span> <span class="p">}</span>
<span class="c1">// ... snip ...</span>
<span class="p">]</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>我们要注意的是<code class="highlighter-rouge">facets.published.entries</code>数组,和上面的例子一样。同样需要一个HTML页来容纳图标和加载数据。机制既然一样,代码就直接看<a href="https://gist.github.com/900542/#file_chart.html">这里</a>吧。</p>
<p>既然已经有了JSON数据,用protovis创建时间线就很简单了,用一个自定义的<a href="http://vis.stanford.edu/protovis/ex/area.html">area chart</a>即可。</p>
<p>完整带注释的<code class="highlighter-rouge">Timeline()</code>函数如下:<br />
```javascript<br />
// =====================================================================================================<br />
// A timeline chart with Protovis - See http://vis.stanford.edu/protovis/ex/area.html<br />
// =====================================================================================================</p>
<p>var Timeline = function(dom_id) {<br />
if (‘undefined’ == typeof dom_id) { // Set the default DOM element ID to bind<br />
dom_id = ‘chart’;<br />
}</p>
<div class="highlighter-rouge"><pre class="highlight"><code>var data = function(json) { // Set the data for the chart
this.data = json;
return this;
};
var draw = function() {
var entries = this.data; // Set-up the data
entries.push({ // Add the last "blank" entry for proper
count : entries[entries.length-1].count // timeline ending
});
// console.log('Drawing, ', entries);
var w = 600, // Set-up dimensions and scales for the chart
h = 100,
max = pv.max(entries, function(d) {return d.count;}),
x = pv.Scale.linear(0, entries.length-1).range(0, w),
y = pv.Scale.linear(0, max).range(0, h);
var vis = new pv.Panel() // Create the basis panel
.width(w)
.height(h)
.bottom(20)
.left(20)
.right(40)
.top(40);
vis.add(pv.Label) // Add the chart legend at top left
.top(-20)
.text(function() {
var first = new Date(entries[0].time);
var last = new Date(entries[entries.length-2].time);
return "Articles published between " +
[ first.getDate(),
first.getMonth() + 1,
first.getFullYear()
].join("/") +
" and " +
[ last.getDate(),
last.getMonth() + 1,
last.getFullYear()
].join("/");
})
.textStyle("#B1B1B1")
vis.add(pv.Rule) // Add the X-ticks
.data(entries)
.visible(function(d) {return d.time;})
.left(function() { return x(this.index); })
.bottom(-15)
.height(15)
.strokeStyle("#33A3E1")
.anchor("right").add(pv.Label) // Add the tick label (DD/MM)
.text(function(d) {
var date = new Date(d.time);
return [
date.getDate(),
date.getMonth() + 1
].join('/');
})
.textStyle("#2C90C8")
.textMargin("5")
vis.add(pv.Rule) // Add the Y-ticks
.data(y.ticks(max)) // Compute tick levels based on the "max" value
.bottom(y)
.strokeStyle("#eee")
.anchor("left").add(pv.Label)
.text(y.tickFormat)
.textStyle("#c0c0c0")
vis.add(pv.Panel) // Add container panel for the chart
.add(pv.Area) // Add the area segments for each entry
.def("active", -1) // Auxiliary variable to hold mouse state
.data(entries) // Pass the data to Protovis
.bottom(0)
.left(function(d) {return x(this.index);}) // Compute x-axis based on scale
.height(function(d) {return y(d.count);}) // Compute y-axis based on scale
.interpolate('cardinal') // Make the chart curve smooth
.segmented(true) // Divide into "segments" (for interactivity)
.fillStyle("#79D0F3")
.event("mouseover", function() { // On "mouse over", set segment as active
this.active(this.index);
return this.root.render();
})
.event("mouseout", function() { // On "mouse out", clear the active state
this.active(-1);
return this.root.render();
})
.event("mousedown", function(d) { // On "mouse down", perform action,
var time = entries[this.index].time; // eg filtering the results...
return (alert("Timestamp: '"+time+"'"));
})
.anchor("top").add(pv.Line) // Add thick stroke to the chart
.lineWidth(3)
.strokeStyle('#33A3E1')
.anchor("top").add(pv.Dot) // Add the circle "label" displaying
// the count for this day
.visible( function() { // The label is only visible when
return this.parent.children[0] // its segment is active
.active() == this.index;
})
.left(function(d) { return x(this.index); })
.bottom(function(d) { return y(d.count); })
.fillStyle("#33A3E1")
.lineWidth(0)
.radius(14)
.anchor("center").add(pv.Label) // Add text to the label
.text(function(d) {return d.count;})
.textStyle("#E7EFF4")
.root.canvas(dom_id) // Bind the chart to DOM element
.render(); // And render it.
};
return { // Create the public API
data : data,
draw : draw
};
</code></pre>
</div>
<p>};<br />
```</p>
<p>完整示例代码在<a href="http://www.elasticsearch.cn/blog/assets/dashboards/timeline.html">这里</a>。不过先去下载protovis提供的关于<a href="http://vis.stanford.edu/protovis/docs/area.html">area</a>的原始文档,然后观察当你修改<code class="highlighter-rouge">interpolate('cardinal')</code>成<code class="highlighter-rouge">interpolate('step-after')</code>后发生了什么。对于多个facet,画叠加的区域图,添加交互性,然后完全定制可视化应该都不是什么问题了。</p>
<p>重要的是注意,这个图表完全是根据你传递给ES的请求做出的响应,使得你有可能做到简单立刻的完成某项指标的可视化需求。比如“显示这个作者在这个主题上最近三个月的出版频率”。只需要提交这样的请求就够了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> author:John AND topic:Search AND published:[2011-03-01 TO 2011-05-31]
</code></pre>
</div>
<h1 id="section-1">总结</h1>
<p>当你需要为复杂的自定义查询做一个丰富的交互式的数据可视化时,使用ES的facets应该是最容易的办法之一,你只需要传递ES的JSON响应给<a href="http://vis.stanford.edu/protovis/">Protovis</a>这样的工具就好了。</p>
<p>通过模仿本文中的方法和代码,你可以在几小时内给你的数据跑通一个示例。</p>
【翻译】Coro模块文档
2012-11-18T00:00:00+08:00
perl
http://chenlinux.com/2012/11/18/coro
<h1 id="section">名称</h1>
<p>Coro —— perl唯一真正的线程</p>
<h1 id="section-1">简要</h1>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nv">Coro</span><span class="p">;</span>
<span class="nv">async</span> <span class="p">{</span>
<span class="c1">#一些异步执行的线程</span>
<span class="k">print</span> <span class="s">"2\n"</span><span class="p">;</span>
<span class="nv">cede</span><span class="p">;</span> <span class="c1">#切换回main线程</span>
<span class="k">print</span> <span class="s">"4\n"</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">print</span> <span class="s">"1\n"</span><span class="p">;</span>
<span class="nv">cede</span><span class="p">;</span> <span class="c1">#切换到coro线程</span>
<span class="k">print</span> <span class="s">"3\n"</span><span class="p">;</span>
<span class="nv">cede</span><span class="p">;</span> <span class="c1">#再次切换</span>
<span class="c1">#使用信号锁</span>
<span class="k">my</span> <span class="nv">$lock</span> <span class="o">=</span> <span class="k">new</span> <span class="nn">Coro::</span><span class="nv">Semaphore</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$locked</span><span class="p">;</span>
<span class="nv">$lock</span><span class="o">-></span><span class="nv">down</span><span class="p">;</span>
<span class="nv">$locked</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="nv">$lock</span><span class="o">-></span><span class="nv">up</span><span class="p">;</span>
</code></pre>
</div>
<h1 id="section-2">描述</h1>
<p>如果想看教程式的介绍,请阅读<a href="http://search.cpan.org/perldoc?Coro::Intro">Coro::Intro</a>文档。这里主要介绍的是一些参考信息。</p>
<p>一般来说,本模块以协作线程(文档中简写成coro)的方式来汇总管理后续代码。他们和内核线程很像但是一般来说不会在SMP机器上同时并发执行。这个模块提供的线程特殊的格式保证了他只在必须的时候才会在你程序中标明了的地方切换线程,所以锁和并发访问都不太会成为问题。Coro模块让线程编程变得更安全,更容易了。</p>
<p>不像那些所谓的“Perl threads”(这并不是真的线程,而是windows进程仿真(详见下面同名章节)移植到UNIX的,实际工作还是进程),Coro提供了一个完全共享的地址空间。这使得线程之间的通信变得非常容易。而且Coro线程也非常快:放弃掉你的perl程序中windows进程仿真的代码改用coro可以很容易的获得2到4倍的速度提升。一个并行矩阵乘法基准测试(高度密集的通信)用coro在单核上运行也比在4核上跑满perl伪线程快300倍。</p>
<p>Coro通过支持多个运行中的解释器共享数据做到的这点。这对写伪并行进程和事件驱动程序非常有用,比如多个HTTP协议的GET请求并发运行。可以看看<a href="http://search.cpan.org/perldoc?Coro::AnyEvent">Coro::AnyEvent</a>模块来学习怎样集成Coro到事件驱动环境里。</p>
<p>在这个模块里,一个县城被定义为“调用链+词法变量+包变量+C栈”。也就是说,一个线程有自己的调用链,自己的词法集,自己的关键性的全局变量(更多配置和背景知识见<a href="http://search.cpan.org/perldoc?Coro::State">Coro::State</a>模块)。</p>
<p>注意看文档结尾的<code class="highlighter-rouge">参考</code>区域——Coro模块的家族可是非常庞大的。</p>
<h1 id="coro">Coro线程生命周期</h1>
<p>在coro线程漫长而兴奋(或许也不)的生命中,它咬经过一系列的状态:</p>
<ul>
<li>1、创建</li>
</ul>
<p>coro线程生命中的第一件事情当然是创建——创建方法就是调用<code class="highlighter-rouge">async块</code>函数:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">async</span> <span class="p">{</span>
<span class="c1">#这里写线程的代码</span>
<span class="p">};</span>
</code></pre>
</div>
<p>你也可以传递参数给代码块,默认会存进<code class="highlighter-rouge">@_</code>里:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">async</span> <span class="p">{</span>
<span class="k">print</span> <span class="nv">$_</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span> <span class="c1">#打印2</span>
<span class="p">}</span> <span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">;</span>
</code></pre>
</div>
<p>这会创建一个新的coro线程并放进ready队列里,这意味着当CPU空闲后它会立刻运行。</p>
<p><code class="highlighter-rouge">async</code>会返回一个Coro对象——你可以把这个对象存起来给之后使用——这个对象就是一个运行中、准备运行或者等待事件中的线程。</p>
<p>另一个创建线程的办法是调用带有代码引用的<code class="highlighter-rouge">new</code>构造器:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">new</span> <span class="nv">Coro</span> <span class="k">sub </span><span class="p">{</span>
<span class="c1">#这里写线程代码</span>
<span class="p">},</span> <span class="nv">@optional_arguments</span><span class="p">;</span>
</code></pre>
</div>
<p>这和调用<code class="highlighter-rouge">async</code>相当类似,唯一的区别就是新线程默认不会放进ready队列里。你不显式的放进去的话,这个线程就永远不会执行了。所以,<code class="highlighter-rouge">async</code>应该等于下面这样的写法:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">my</span> <span class="nv">$coro</span> <span class="o">=</span> <span class="k">new</span> <span class="nv">Coro</span> <span class="k">sub </span><span class="p">{</span>
<span class="c1">#这里写线程代码</span>
<span class="p">};</span>
<span class="nv">$coro</span><span class="o">-></span><span class="nv">ready</span><span class="p">;</span>
<span class="k">return</span> <span class="nv">$coro</span><span class="p">;</span>
</code></pre>
</div>
<ul>
<li>2、启动</li>
</ul>
<p>当新coro线程创建之后,只会保存一个代码引用和参数。并不立刻分配额外的内存给栈。这样可以保持coro线程的低内存使用水平。</p>
<p>只有当线程真正开始运行的时候,这些资源才会分配出来。</p>
<p>附加参数在coro创建的时候存进了<code class="highlighter-rouge">@_</code>里,这点和函数调用是一样的。</p>
<ul>
<li>3、运行、阻塞</li>
</ul>
<p>当coro线程开始运行之后会发生很多事情。一般来说,它不会一口气跑到底(因为这种情况你肯定是直接用普通函数代替了),它会让出CPU来等待其他外部事件。</p>
<p>只要coro线程还在运行,这个Coro对象就一直存在一个叫做<code class="highlighter-rouge">$Coro::current</code>的全局变量里。</p>
<p>一个底层的让出CPU的办法是调用调度器,调度器会选择一个新的线程来运行:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nn">Coro::</span><span class="nv">schedule</span><span class="p">;</span>
</code></pre>
</div>
<p>因为运行中的线程不可能在ready队列里,所以啥都不做单纯调用调度器会永远的阻塞住coro线程——你必须安排好由某些事件或者线程唤醒coro线程,或者是在调度前直接把coro线程放进ready队列:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="c1">#这其实就是Coro::cede做的</span>
<span class="nv">$</span><span class="nn">Coro::</span><span class="nv">current</span><span class="o">-></span><span class="nv">ready</span><span class="p">;</span>
<span class="nn">Coro::</span><span class="nv">schedule</span><span class="p">;</span>
</code></pre>
</div>
<p>所有高级的同步方法(Coro::Semaphore, Coro::rouse_*)都是通过<code class="highlighter-rouge">->ready</code>和<code class="highlighter-rouge">Coro::schedule</code>来实现的。</p>
<p>当coro线程运行的时候,它可能被分配到一个C级别的线程,也可能从C级别的线程里被剥离,一切如Coro运行时所愿。当你的perl线程调用C级别的函数的时候,就需要分配到C线程,然后这个函数反过来调用perl,然后perl就要想办法切换协程。在你运行事件循环然后在回调中阻塞的时候经常会出现这个情况。还有一种情况是perl自己通过<code class="highlighter-rouge">tie</code>机制调用一些方法或者函数比如<code class="highlighter-rouge">AUTOLOAD</code>等等。</p>
<ul>
<li>4、终止</li>
</ul>
<p>一段时间后,大多数线程都会终止。有很多办法来终止一个coro线程。最简单的就是从顶级代码引用里返回:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">async</span> <span class="p">{</span>
<span class="c1">#当从这里return后,coro线程自然就终止了</span>
<span class="p">};</span>
<span class="nv">async</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">if</span> <span class="mf">0.5</span> <span class="o"><</span> <span class="nb">rand</span><span class="p">;</span> <span class="c1">#可能提前从这里就终止了</span>
<span class="k">print</span> <span class="s">"got a chance to print this\n"</span><span class="p">;</span>
<span class="c1">#或者在这里终止</span>
<span class="p">};</span>
</code></pre>
</div>
<p>从协程里返回的任意值都可以由<code class="highlighter-rouge">->join</code>获取:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">my</span> <span class="nv">$coro</span> <span class="o">=</span> <span class="nv">async</span> <span class="p">{</span>
<span class="s">"hello, world\n"</span> <span class="c1">#返回一个字符串</span>
<span class="p">}</span>
<span class="k">my</span> <span class="nv">$hello_world</span> <span class="o">=</span> <span class="nv">$coro</span><span class="o">-></span><span class="nb">join</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">$hello_world</span><span class="p">;</span>
</code></pre>
</div>
<p>另一个办法就是调用<code class="highlighter-rouge">Coro::terminate</code>方法,在任意嵌套级别的子例程里都行:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">async</span> <span class="p">{</span>
<span class="nn">Coro::</span><span class="nv">terminate</span> <span class="s">"return value 1"</span><span class="p">,</span> <span class="s">"return value 2"</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
<p>还有一个办法是从另一个线程<code class="highlighter-rouge">->cancel</code>(或者<code class="highlighter-rouge">->safe_cancel</code>)一个coro线程:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">my</span> <span class="nv">$coro</span> <span class="o">=</span> <span class="nv">async</span> <span class="p">{</span>
<span class="nb">exit</span> <span class="mi">1</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">$coro</span><span class="o">-></span><span class="nv">cancel</span><span class="p">;</span> <span class="c1">#同样接收数据给->join获取</span>
</code></pre>
</div>
<p>取消操作通常<code class="highlighter-rouge">可能</code>会很危险——它有点像调用了<code class="highlighter-rouge">exit</code>却又没真的退出。然后可能把C库和XS模块遗留在一个古怪的状态。而且和其他的线程实现不一样的是,Coro关于取消方面的异常是安全的。在你想用Coro做些奇妙的事情而又被取消的情况下,Perl会一直保持一个一致的状态——那就是,确保线程被取消的时候,所有清理代码都被执行了——所以还有一个<code class="highlighter-rouge">->safe_cancel</code>方法。</p>
<p>所以,在一个XS的事件循环里取消一个线程可能不是最好的主意。不过在只有perl(比如<code class="highlighter-rouge">tie</code>方法或<code class="highlighter-rouge">AUTOLOAD</code>)的其他组合里这么处理是安全的。<br />
最后,Coro线程对象在<code class="highlighter-rouge">->cancel</code>后自动的被取消引用了——和Perl里其他对象一样。虽然这不是什么普遍的情况,一个运行中的线程被<code class="highlighter-rouge">$Coro::current</code>引用,一个等待运行中的线程被ready队列引用,一个等待锁或者信号的线程被等待列表引用,等等等等……但取消的时候,所有队列都不再有这个线程了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">async</span> <span class="p">{</span>
<span class="nv">schedule</span><span class="p">;</span> <span class="c1">#切换到其他coro里,不进ready队列</span>
<span class="p">};</span>
<span class="nv">cede</span><span class="p">;</span>
<span class="c1">#现在上面的async被摧毁了,不再被任何地方引用。</span>
</code></pre>
</div>
<ul>
<li>5、清理</li>
</ul>
<p>线程需要分配各种资源。大多数但不是所有在线程终止的时候会被清理返回。</p>
<p>清理非常像丢弃未捕获的异常:perl会按照它的方式去运行所有的子例程调用和代码块。它的方式里,它会释放所有的<code class="highlighter-rouge">my</code>变量,撤销所有的<code class="highlighter-rouge">local</code>变量,释放所有其他线程独立的资源。</p>
<p>所以,常见的释放资源的办法就是让他们成为my变量。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">async</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$big_cache</span> <span class="o">=</span> <span class="k">new</span> <span class="nv">Cache</span> <span class="o">...</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<p>如果不再有引用存在,<code class="highlighter-rouge">$big_cache</code>对象在线程终止的时候自然就释放掉了。</p>
<p>它并<code class="highlighter-rouge">不</code>会解锁Coro::Semaphore或类似的其他资源,这时候<code class="highlighter-rouge">guard</code>方法就派上用场了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">my</span> <span class="nv">$sem</span> <span class="o">=</span> <span class="k">new</span> <span class="nn">Coro::</span><span class="nv">Semaphore</span><span class="p">;</span>
<span class="nv">async</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$lock_guard</span> <span class="o">=</span> <span class="nv">$sem</span><span class="o">-></span><span class="nv">guard</span><span class="p">;</span>
<span class="c1">#如果我们在这里return,或者die,或者取消</span>
<span class="c1">#信号也就唤醒了</span>
<span class="p">}</span>
</code></pre>
</div>
<p>这个<code class="highlighter-rouge">Guard::guard</code>函数可以在你想要的时候出现在任意清理的时候(但是不能从代码块里切换到其他协程中):</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">async</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$window</span> <span class="o">=</span> <span class="k">new</span> <span class="nn">Gtk2::</span><span class="nv">window</span> <span class="s">"toplevel"</span><span class="p">;</span>
<span class="c1">#window不会被自动清理,哪怕$window被释放了</span>
<span class="c1">#所以用guard确保在出错的时候它可以被正确的毁灭</span>
<span class="k">my</span> <span class="nv">$window_guard</span> <span class="o">=</span> <span class="nn">Guard::</span><span class="nv">guard</span> <span class="p">{</span><span class="nv">$window</span><span class="o">-></span><span class="nv">destroy</span><span class="p">};</span>
<span class="c1">#这样从这里开始我就安全了</span>
<span class="p">};</span>
</code></pre>
</div>
<p>最后,<code class="highlighter-rouge">local</code>通常也是很方便的。比如临时替换一下coro线程的描述:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">sub </span><span class="nf">myfunction</span> <span class="p">{</span>
<span class="nb">local</span> <span class="nv">$</span><span class="nn">Coro::</span><span class="nv">current</span><span class="o">-></span><span class="p">{</span><span class="nv">desc</span><span class="p">}</span> <span class="o">=</span> <span class="s">"inside myfunction(@_)"</span><span class="p">;</span>
<span class="c1">#如果这里突然return或者die了,描述会重新存储过</span>
<span class="p">}</span>
</code></pre>
</div>
<ul>
<li>6、僵尸死亡万岁</li>
</ul>
<p>即便一个线程已经终止并且清理过它的资源了,Coro对象依然存在,而且存储着它的线程返回值。</p>
<p>这意味着线程终止并清理,不再有其他引用之后,Coro对象会自动释放掉。</p>
<p>而如果还有引用,Coro对象就还保留着,你可以调用<code class="highlighter-rouge">->join</code>多次来接收结果数据:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">async</span> <span class="p">{</span>
<span class="k">print</span> <span class="s">"hi\n"</span><span class="p">;</span>
<span class="mi">1</span>
<span class="p">};</span>
<span class="c1">#运行上面的async,并且在从Coro::cede返回前释放所有资源</span>
<span class="nn">Coro::</span><span class="nv">cede</span><span class="p">;</span>
<span class="p">{</span>
<span class="k">my</span> <span class="nv">$coro</span> <span class="o">=</span> <span class="nv">async</span> <span class="p">{</span>
<span class="k">print</span> <span class="s">"hi\n"</span><span class="p">;</span>
<span class="mi">1</span>
<span class="p">};</span>
<span class="c1">#运行上面的async并清理掉,但是不是放coro对象:</span>
<span class="nn">Coro::</span><span class="nv">cede</span><span class="p">;</span>
<span class="c1">#可选的收取结果</span>
<span class="k">my</span> <span class="nv">@results</span> <span class="o">=</span> <span class="nv">$coro</span><span class="o">-></span><span class="nb">join</span><span class="p">;</span>
<span class="c1">#现在$coro超出范围了,可能被释放掉</span>
<span class="p">};</span>
</code></pre>
</div>
<h1 id="section-3">全局变量</h1>
<ul>
<li>$Coro::main</li>
</ul>
<p>这个变量存储了代表主程序的Coro对象。如果你可以<code class="highlighter-rouge">ready</code>好它,可以像操作coro一样操作它。在对比<code class="highlighter-rouge">$Coro::current</code>的时候特别有用,这样可以看到自己是不是运行在主程序里了。</p>
<ul>
<li>$Coro::current</li>
</ul>
<p>这个变量代表当前coro(Coro调度器切换到的最后一个coro)。初始值和<code class="highlighter-rouge">$Coro::main</code>一样。</p>
<p>这个变量是__严格___只读_的。你可以复制到别的变量然后在其他Coro对象里使用,但不能修改这个变量本身。</p>
<ul>
<li>$Coro::idle</li>
</ul>
<p>这个变量在集成Coro到事件循环的时候很有用。通常他更依赖<a href="http://search.cpan.org/perldoc?Coro::AnyEvent">Coro::AnyEvent</a>或者<a href="http://search.cpan.org/perldoc?Coro::EV">Coro::EV</a>,这是很漂亮的底层功能。</p>
<p>这个变量存储的Coro对象在没有其他ready线程的时候,就会被放进ready队列里(而不会调用其他ready钩子)。</p>
<p>默认实现是带着一个“致命的:检测到死锁”的提示die退出,然后跟着线程列表,因为程序没办法继续了。</p>
<p>钩子被<code class="highlighter-rouge">Coro::EV</code>和<code class="highlighter-rouge">Coro::AnyEvent</code>这样的模块重写以等待外部事件唤醒coro以便调度器运行。</p>
<p>这个技术的示例请参见<a href="http://search.cpan.org/perldoc?Coro::EV">Coro::EV</a>或者<a href="http://search.cpan.org/perldoc?Coro::AnyEvent">Coro::AnyEvent</a>模块。</p>
<h1 id="coro-1">简单的Coro创建</h1>
<ul>
<li>async {…} [@args…]</li>
</ul>
<p>创建新coro返回他的Coro对象(通常用不上)。这个coro会被放进ready队列,当下一次调度来临的时候就自动运行。</p>
<p>第一个参数是要在coro里运行的代码块/闭包。当它返回时,coro自动终止。</p>
<p>剩余参数作为闭包的参数传递进去。</p>
<p>参见<code class="highlighter-rouge">Coro::State::new</code>构造器来了解当coro运行时coro环境的信息。</p>
<p>在coro里调用<code class="highlighter-rouge">exit</code>和在外头的效果是一样的,同样,如果coro线程die掉,程序整个退出,和在cor外面也一样。</p>
<p>如果你不想这样,你可以通过一个默认的<code class="highlighter-rouge">die</code>句柄,或者简单的用<code class="highlighter-rouge">eval</code>包装一下。</p>
<p>示例:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">async</span> <span class="p">{</span>
<span class="k">print</span> <span class="s">"@_\n"</span><span class="p">;</span>
<span class="p">}</span> <span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">;</span>
</code></pre>
</div>
<ul>
<li>async_pool {…} [@args…]</li>
</ul>
<p>和<code class="highlighter-rouge">async</code>类似,不过用一个coro池,所以你不要对这个对象调用terminate或者join方法(然后我们也没禁止)。而且你可能得到一个coro是已经在执行其他代码的(这事儿说好也好,说不好也不好)。</p>
<p>从加强的的一面说,这个函数比完整的创建(和销毁)一个新coro快了两倍。所以你如果需要快速创建大批量的通用coro,使用<code class="highlighter-rouge">async_pool</code>,别用<code class="highlighter-rouge">async</code>。</p>
<p>代码块会在<code class="highlighter-rouge">eval</code>环境里运行,出现异常的时候抛出warning而不是终止程序,这和<code class="highlighter-rouge">async</code>一样。</p>
<p>当coro被重用的时候,像<code class="highlighter-rouge">on_destroy</code>这样的东西可能不会按照你想象的那样工作,除非你调用终止或者取消,这些都是跟池的目的相违背的(不过在异常的情况下还是很不错的)。</p>
<p>每次运行后,优先级都设置成<code class="highlighter-rouge">0</code>,跟踪被禁用,描述被清空,默认输出句柄被恢复。这样你可以改变所有这些东西。否则,coro会重用他们的“初始值”:最显著的就是如果你修改了每个线程的全局变量比如<code class="highlighter-rouge">$/</code>,你必须修复这个改变。最简单的做法就是用<code class="highlighter-rouge">local $/</code>这样。</p>
<p>空闲池的大小限定为<code class="highlighter-rouge">8</code>个空闲线程(这可以通过$Coro::POOL_SIZE改变),但是有需求的恶化,非空闲的coro是多多益善的。</p>
<p>如果一个<code class="highlighter-rouge">async_pool</code>用了太多栈空间让你担心池里的coro章太猛了,你可以每秒钟运行<code class="highlighter-rouge">async_pool {terminate}</code>这样的代码来缓慢的补充池子。除此之外,当句柄用的栈涨到超过32KB(由$Coro::POOL_RSS设置)时,它就会被销毁。</p>
<h1 id="section-4">静态方法</h1>
<p>静态方法实际上就是对当前coro进行隐式操作的函数。</p>
<ul>
<li>schedule</li>
</ul>
<p>调用调度器。调度器会从ready队列中查找下一个可以运行的coro并切换过去。这个“下一个可以运行的coro”就是有最高优先级的,在队列里等待时间最久的那个。如果一个都没有,就调用<code class="highlighter-rouge">$Coro::idle</code>钩子。</p>
<p>请注意:当前coro<code class="highlighter-rouge">不</code>会被放进ready队列里,所以调用这个函数后,这个coro不再会被调用知道有其他事件调用<code class="highlighter-rouge">->ready</code>来唤醒你。</p>
<p>这让<code class="highlighter-rouge">schedule</code>阻塞当前线程并等待事件:首先你要把当前coro记在一个变量里,然后安排好回调,在某些情况下可以用<code class="highlighter-rouge">->ready</code>来唤醒你,最后你调用<code class="highlighter-rouge">schedule</code>让自己进入沉睡。注意有很多办法可以唤醒coro,所以你要检测一下事件是否正确,比如把状态存储在一个变量里。</p>
<p>至于怎样等待回调,参见下面的__怎么等待回调__章节。</p>
<ul>
<li>cede</li>
</ul>
<p>“放弃”到其他coro。这个函数把当前线程放进ready队列里然后调用<code class="highlighter-rouge">schedule</code>。它的效果是放弃当前的“时间片”给其他拥有更高优先级或者至少同级别的coro。一旦你的coro重新被轮到,它会自动恢复过来。</p>
<p>在其他语言里,这个函数经常被叫做<code class="highlighter-rouge">yield</code>。</p>
<ul>
<li>Coro::cede_notself</li>
</ul>
<p>和cede类似,不过默认不会export出来,这个函数会不顾优先级强制cede给_其他_coro,在需要确保进程运行的时候还是有些用的。</p>
<ul>
<li>terminate [arg…]</li>
</ul>
<p>带着给定的状态值终止当前coro(参见<a href="http://search.cpan.org/perldoc?cancel">cancel</a>)。这些状态值不会被直接返回,而是返回他们的引用。</p>
<ul>
<li>Coro::on_enter BLOCK, Coro::on_leave BLOCK</li>
</ul>
<p>这两函数会在当前作用域内安装enter和leave。enter块会在on_enter 被调用,还有当前coro被调度器re-enter的时候执行。而leave快则是在当前coro被调度器阻塞,还有词法作用域被退出(意思就是exit、die、last等)的时候被执行。</p>
<p><em>在这些块里,不允许再调用调度器,也不允许异常</em>。这意味着,不用eval的情况下别想调用<code class="highlighter-rouge">die</code>命令,至于调度器更是什么办法都没法用了。</p>
<p>介于这些块都是和当前作用域绑定的,所以当当前作用域退出的时候,他们会自动删除。</p>
<p>这两函数实现了和计划中<code class="highlighter-rouge">dynamic-wind</code>做的一样的概念,在你想给一个特定coro本地化某些资源的时候比较有用。</p>
<p>使用这两函数的coro会相对的被放慢线程切换的速度(大概一个单独分配的块40%的样子,所以只要处理程序够快,线程切换依然很快)。</p>
<p>通过下面这个例子,可以更好的理解这些函数:切换当前时区到”南极”,这需要调用<code class="highlighter-rouge">tzset</code>,但是我们使用<code class="highlighter-rouge">on_enter</code>和<code class="highlighter-rouge">on_leave</code>,用来记忆/改变当前时区并存储之前的值。分别的,只有安装了这两函数的coro才会改变时区。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nv">POSIX</span> <span class="sx">qw(tzset)</span><span class="p">;</span>
<span class="nv">async</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$old</span><span class="o">\</span><span class="nv">_tz</span><span class="p">;</span> <span class="o">\</span><span class="c1">#在这里存储外面的时区</span>
<span class="nn">Coro::</span><span class="nv">on</span><span class="o">\</span><span class="nv">_enter</span> <span class="p">{</span>
<span class="nv">$old</span><span class="o">\</span><span class="nv">_tz</span> <span class="o">=</span> <span class="nv">$ENV</span><span class="p">{</span><span class="nv">TZ</span><span class="p">};</span> <span class="o">\</span><span class="c1">#记忆旧的数值</span>
<span class="nv">$ENV</span><span class="p">{</span><span class="nv">TZ</span><span class="p">}</span> <span class="o">=</span> <span class="s">"Antarctica/South\_Pole"</span><span class="p">;</span>
<span class="nv">tzset</span><span class="p">;</span> <span class="o">\</span><span class="c1">#启用新值</span>
<span class="p">};</span>
<span class="nn">Coro::</span><span class="nv">on</span><span class="o">\</span><span class="nv">_leave</span> <span class="p">{</span>
<span class="nv">$ENV</span><span class="p">{</span><span class="nv">TZ</span><span class="p">}</span> <span class="o">=</span> <span class="nv">$old</span><span class="o">\</span><span class="nv">_tz</span><span class="p">;</span>
<span class="nv">tzset</span><span class="p">;</span> <span class="o">\</span><span class="c1">#恢复旧值</span>
<span class="p">};</span>
<span class="o">\</span><span class="c1">#在这块,时区就是"南极",不会被其他coro里的时区影响</span>
<span class="p">};</span>
</code></pre>
</div>
<p>这可以用于给块本地化任何资源(locale,uid,当前工作目录等),尽管当前有其他coro存在。</p>
<p>另一个有趣的例子,通过间隔计时器实现了时间片的多任务(下面的代码明显是可以优化的,不过当前足够跑任务了):</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="o">\</span><span class="c1">#把给定块按时间分片</span>
<span class="k">sub </span><span class="nf">timeslice</span><span class="p">(&) {</span>
<span class="k">use</span> <span class="nn">Time::</span><span class="nv">HiRes</span> <span class="p">();</span>
<span class="nn">Coro::</span><span class="nv">on</span><span class="o">\</span><span class="nv">_enter</span> <span class="p">{</span>
<span class="o">\</span><span class="c1">#在进线程的时候,我们设置一个VTALRM信号以便cede</span>
<span class="nv">$SIG</span><span class="p">{</span><span class="nv">VTALRM</span><span class="p">}</span> <span class="o">=</span> <span class="k">sub </span><span class="p">{</span> <span class="nv">cede</span> <span class="p">};</span>
<span class="o">\</span><span class="c1">#然后启动一个间隔计时器</span>
<span class="nn">Time::HiRes::</span><span class="nv">setitimer</span> <span class="o">&</span><span class="nn">Time::HiRes::</span><span class="nv">ITIMER</span><span class="o">\</span><span class="nv">_VIRTUAL</span><span class="p">,</span> <span class="mf">0.01</span><span class="p">,</span> <span class="mf">0.01</span><span class="p">;</span>
<span class="p">};</span>
<span class="nn">Coro::</span><span class="nv">on</span><span class="o">\</span><span class="nv">_leave</span> <span class="p">{</span>
<span class="o">\</span><span class="c1">#在离开线程的时候我们停止这个间隔计时器</span>
<span class="nn">Time::HiRes::</span><span class="nv">setitimer</span> <span class="o">&</span><span class="nn">Time::HiRes::</span><span class="nv">ITIMER</span><span class="o">\</span><span class="nv">_VIRTUAL</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">};</span>
<span class="o">&</span><span class="p">{</span><span class="o">+</span><span class="nb">shift</span><span class="p">};</span>
<span class="p">}</span>
<span class="o">\</span><span class="c1">#使用方法如下:</span>
<span class="nv">timeslice</span> <span class="p">{</span>
<span class="o">\</span><span class="c1">#下面是一个死循环,一般情况下会垄断进程。</span>
<span class="o">\</span><span class="c1">#不过现在它跑着一个时间片环境里,定期的会cede给其他线程。</span>
<span class="k">while</span> <span class="p">()</span> <span class="p">{</span> <span class="p">}</span>
<span class="p">};</span>
</code></pre>
</div>
<ul>
<li>killall</li>
</ul>
<p>除当前运行的coro外,杀死/中断/取消所有coro。</p>
<p>注意,如果调用killall的coro不是主coro,当他试图释放一些主解释资源的时候,可能释放不干净。会存在一些一次性的资源泄露。</p>
<h1 id="coro-2">Coro对象方法</h1>
<p>下面是一些你可以在coro对象上调用(或者创建)的方法。</p>
<ul>
<li>new Coro \&sub [,@args…]</li>
</ul>
<p>创建一个新的coro并返回它。当sub返回的时候,coro自动终止,就像你带着返回值调用<code class="highlighter-rouge">terminate</code>的效果一样。要让coro运行,你要先调用rady方法把它放进ready队列里。</p>
<p>参考<code class="highlighter-rouge">async</code>和<code class="highlighter-rouge">Coro::State::new</code>查看更多关于coro环境的信息。</p>
<ul>
<li>$success = $coro->ready</li>
</ul>
<p>将该coro放进它的ready队列的最后(每个优先级都有一个队列)并返回真。如果coro已经在ready队列里,不做任何操作并返回假。</p>
<p>这保证里当所有高优先级的coro和同优先级先准备好了的coro都恢复后,调度器会自动恢复这个coro。</p>
<ul>
<li>$coro->suspend</li>
</ul>
<p>挂起指定coro。一个挂起的coro和其他coro一样工作,不同的是调度器不会选择挂起的coro做真正的执行。</p>
<p>当你想阻止某个coro运行又不打算销毁它,或者当你想暂时冻结某个coro(比方需要调试)等之后再恢复的时候,挂起就很有用了。</p>
<p>前者的一个场景可能是这样:fork之后挂起所有其他的coro但保持住他们不调用析构器,不过你可以继续创建新的coro。</p>
<ul>
<li>$coro->resume</li>
</ul>
<p>当指定coro被挂起后,它就可以被恢复。注意如果一个已经在ready队列里的coro被挂起,调度器可能会把它踢出去,你会失去这次激活。</p>
<p>要避免这种情况的话,最好的办法是无条件的把挂起coro放进预备队列,每个同步机制都必然会保护自己不被虚假唤醒,Coro自然也有。</p>
<ul>
<li>$state->is_new</li>
</ul>
<p>如果Coro对象还是“新”的,返回真,额,新的意思是还没运行过。这些状态基本只是由要调用的代码引用和参数组成。消耗的其他资源很少。转移到新状态后会自动分配一个perl解释器。</p>
<ul>
<li>$state->is_zombie</li>
</ul>
<p>如果Coro对象被取消了,返回真。比如对象的资源因为<code class="highlighter-rouge">cancel</code>、<code class="highlighter-rouge">terminate</code>、<code class="highlighter-rouge">safe_cancel</code>释放了,或者可能就是简单的跑出范围了。</p>
<p>“僵尸”这个名字源自UNIX文化,当一个进程已经退出,除了退出状态什么资源都没有了的时候,就会被叫做“僵尸”。</p>
<ul>
<li>$is_ready = $coro->is_ready</li>
</ul>
<p>如果Coro对象在预备队列里,返回真。它最终会被调度器调控,除非Coro对象被销毁。</p>
<ul>
<li>$is_running = $coro->is_running</li>
</ul>
<p>如果Coro对象正在运行,返回真。只有一个Coro对象可以处于运行状态(但一个Coro对象可以有多个运行中的Coro::States)。</p>
<ul>
<li>$is_suspended = $coro->is_suspended</li>
</ul>
<p>如果Coro对象被挂起,返回真。挂起的Coro永远不会被调度。</p>
<ul>
<li>$coro->cancel (arg…)</li>
</ul>
<p>终止指定Coro线程,强制返回指定参数作为状态(默认为空列表)。如果指定Coro就是当前Coro,则无法返回。</p>
<p>这是一个相当残酷的释放coro的方式,而且还有一些限制——如果线程里有一个不希望被终止的C语言的回调,有些不忍言之事就要发生了;或者如果取消的线程上运行着复杂的清理程序,而这个清理程序又依赖于它的线程上下文,事情也不大会正常的工作。</p>
<p>要运行的清理程序代码(比如<code class="highlighter-rouge">guard</code>代码块)不会有线程上下文,也不允许再切换到其他线程。</p>
<p>另外,<code class="highlighter-rouge">->cancel</code>永远都是这么不管不顾的清理线程。所以如果你的清理代码很复杂或者你希望避免取消一个自己压根不知道怎么清理的C语言线程,建议使用<code class="highlighter-rouge">->throw</code>抛出异常,或者用<code class="highlighter-rouge">->safe_cancel</code>方法。</p>
<p>传递给<code class="highlighter-rouge">->cancel</code>的参数不会被复制,而是被直接引用(比如:你传递了<code class="highlighter-rouge">$var</code>,在调用修改这个变量之后,你也需要修改传递给<code class="highlighter-rouge">join</code>的返回值,所以最好别用这个)。</p>
<p>Coro的资源通常在这个调用返回之前就已经都释放或销毁掉了。不过这事可以被无限期的推迟,因为可能作为管理端的线程有时候要首先运行注销Coro对象。</p>
<ul>
<li>$coro->safe_cancel($arg…)</li>
</ul>
<p>和<code class="highlighter-rouge">->cancel</code>很像。不过本质上,它是“安全”的。所以当线程并不处于一个可终止的状态的时候,它会抛出一个异常。</p>
<p>这个方法运行起来就像抛出一个不可被捕捉的异常——具体的说,它从线程的内部开始清理,所以所有的清理程序(比如<code class="highlighter-rouge">guard</code>块),都是在线程的上下文中运行,并且可以随意阻塞。它的缺点就是不保证线程肯定可以终止,它可能会失败。而且,运行速度也比<code class="highlighter-rouge">cancel</code>和<code class="highlighter-rouge">terminal</code>慢。</p>
<p>一个线程,当它还没有被运行,或者没有C语言的上下文附加且在SLF函数内。</p>
<p>后面这两个的意思基本上就是线程不在被某些C函数(通常是XS模块)回调的perl函数里,也不在这些C函数通过Coro的XS级别的API调用运行中。</p>
<p>当本函数可以正常终止线程时,返回真;否则报错(即要么返回真要么不返回)。</p>
<p>为什么搞这么奇怪的接口?嗯,关于何时如何终止线程,有两种通用模式。一种是你希望当你想终止的时候就可以终止——当线程不可终止的时候,显然就会有问题了。所以需要<code class="highlighter-rouge">->safe_cancel</code>来报错。</p>
<p>第二种模式是,你很友好的问下先,如果不碰巧,那就先不终止线程了。看起来就像这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">if</span> <span class="p">(</span><span class="o">!</span> <span class="nb">eval</span> <span class="p">{</span> <span class="nv">$coro</span><span class="o">-></span><span class="nv">safe</span><span class="o">\</span><span class="nv">_cancel</span> <span class="p">})</span> <span class="p">{</span>
<span class="nb">warn</span> <span class="s">"unable to cancel thread: $@"</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
<p>然而,你不应该总是先尝试安全的取消然后失败了再强行<code class="highlighter-rouge">->cancel</code>。这样是没道理的:因为你肯定要不就在线程里自己搞定清理代码,要不就是没有。有的话,用<code class="highlighter-rouge">->safe_cancel</code>;没有的话,<code class="highlighter-rouge">->cancel</code>更直接快捷。</p>
<ul>
<li>$coro->schedule_to</li>
</ul>
<p>让当前线程进入休眠(类似<a href="http://search.cpan.org/perldoc?Coro::schedule">Coro::schedule</a>),不过不会轮到ready队列的下一个线程,而是切换到给定的那个Coro对象(不管多少优先级)。coro的准备情况并不会被改变。</p>
<p>这是一个为特殊情况准备的高级方法——我很乐意听到它被实际运用了。</p>
<ul>
<li>$coro->cede_to</li>
</ul>
<p>和<code class="highlighter-rouge">schedule_to</code>类似,但是是把当前线程放进ready队列里。它等效于暂时切换到给定的对象,过会儿再继续。</p>
<p>这是一个为特殊情况准备的高级方法——我很乐意听到它被实际运用了。</p>
<ul>
<li>$coro->throw ([$scalar])</li>
</ul>
<p>如果<code class="highlighter-rouge">$throw</code>被定义了,那它会在下一个合适的时间点被coro作为异常抛出。否则就清理掉这个异常对象。</p>
<p>Coro会在每个类schedule函数返回时检查异常。这类函数包括<code class="highlighter-rouge">schedule</code>,<code class="highlighter-rouge">cede</code>,<code class="highlighter-rouge">Coro::Semaphore->down</code>,<code class="highlighter-rouge">Coro::Handle->readable</code>等等。大多数这些函数(都是Coro的一部分)检测这个情况,并且在异常pending的时候提前返回。</p>
<p>异常对象会在<code class="highlighter-rouge">$@</code>中和另一个特殊标量一起被抛出。即,如果它是字符串,不会有行号和和新行追加进来(跟<code class="highlighter-rouge">die</code>不一样)。</p>
<p>这可以被用来作为一个比<code class="highlighter-rouge">cancel</code>或者<code class="highlighter-rouge">safe_cancel</code>更柔和一些的询问一个coro是否结束的办法,虽然并不能保证异常一定会导致终止而且如果没有被捕获它可能会结束整个程序。</p>
<p>你也可以理解<code class="highlighter-rouge">throw</code>是类似带信号(这种情况就是一个标量)的<code class="highlighter-rouge">kill</code>。</p>
<ul>
<li>$coro->join</li>
</ul>
<p>等待coro中止并返回线程给<code class="highlighter-rouge">terminal</code>或<code class="highlighter-rouge">cancel</code>返回的任意值。<code class="highlighter-rouge">join</code>可以并发的被多个线程调用。然后一旦<code class="highlighter-rouge">$coro</code>中止,一切都会恢复并且给出一个返回值。</p>
<ul>
<li>$coro->on_destroy (\&cb)</li>
</ul>
<p>注册一个回调函数在coro线程被销毁的时候被调用。具体的说是资源已经被释放,不过join还没开始。在任意情况下,只要<code class="highlighter-rouge">不</code>是die,这个回调函数都会传入终止/中止参数。</p>
<p>每个coro可以有任意多个<code class="highlighter-rouge">on_destroy</code>回调,而且目前为止,一旦添加,不可以再删除了。</p>
<ul>
<li>$oldprio = $coro->prio ($newprio)</li>
</ul>
<p>设置(当没有参数的时候就是获取)coro线程的优先级。高优先级的会比低优先级的更早运行。优先级是有符号整数,目前是3到-4之间。你可以参考使用PRIO_***常量(提前导入标签:prio获取);</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">PRIO</span><span class="o">\</span><span class="nv">_MAX</span> <span class="o">></span> <span class="nv">PRIO</span><span class="o">\</span><span class="nv">_HIGH</span> <span class="o">></span> <span class="nv">PRIO</span><span class="o">\</span><span class="nv">_NORMAL</span> <span class="o">></span> <span class="nv">PRIO</span><span class="o">\</span><span class="nv">_LOW</span> <span class="o">></span> <span class="nv">PRIO</span><span class="o">\</span><span class="nv">_IDLE</span> <span class="o">></span> <span class="nv">PRIO</span><span class="o">\</span><span class="nv">_MIN</span>
<span class="mi">3</span> <span class="o">></span> <span class="mi">1</span> <span class="o">></span> <span class="mi">0</span> <span class="o">></span> <span class="o">-</span><span class="mi">1</span> <span class="o">></span> <span class="o">-</span><span class="mi">3</span> <span class="o">></span> <span class="o">-</span><span class="mi">4</span>
<span class="o">\</span><span class="c1"># 设置优先级为高</span>
<span class="nv">current</span><span class="o">-></span><span class="nv">prio</span> <span class="p">(</span><span class="nv">PRIO</span><span class="o">\</span><span class="nv">_HIGH</span><span class="p">);</span>
</code></pre>
</div>
<p>空闲的coro线程永远比其他存活的coro优先级要低。</p>
<p>修改当前coro的优先级即时生效,但是修改ready队列里的只会在下次调度(到它)的时候才生效。或者这算个bug,未来某个版本会修正。</p>
<ul>
<li>$newprio = $coro->nice ($change)</li>
</ul>
<p>类似<code class="highlighter-rouge">prio</code>方法,不过是从优先级中减去给定的值(也就是说值越大优先级越低,类似UNIX里的nice命令)。</p>
<ul>
<li>$olddesc = $coro->desc ($newdesc)</li>
</ul>
<p>设置(当没有参数的时候就是获取)coro线程的描述。这只是与coro关联的无格式的字符串。</p>
<p>这个方法只是简单的把<code class="highlighter-rouge">$coro->{desc}</code>成员设置为给定的字符串。你也可以自己修改这个成员。事实上,大家通常宁愿这样声明,比如在一个<a href="http://search.cpan.org/perldoc?Coro::Debug">Coro::Debug</a>的会话里:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">sub </span><span class="nf">my</span><span class="p">\_long\_function {</span>
<span class="nb">local</span> <span class="nv">$</span><span class="nn">Coro::</span><span class="nv">current</span><span class="o">-></span><span class="p">{</span><span class="nv">desc</span><span class="p">}</span> <span class="o">=</span> <span class="s">"now in my\_long\_function"</span><span class="p">;</span>
<span class="o">...</span>
<span class="nv">$</span><span class="nn">Coro::</span><span class="nv">current</span><span class="o">-></span><span class="p">{</span><span class="nv">desc</span><span class="p">}</span> <span class="o">=</span> <span class="s">"my\_long\_function: phase 1"</span><span class="p">;</span>
<span class="o">...</span>
<span class="nv">$</span><span class="nn">Coro::</span><span class="nv">current</span><span class="o">-></span><span class="p">{</span><span class="nv">desc</span><span class="p">}</span> <span class="o">=</span> <span class="s">"my\_long\_function: phase 2"</span><span class="p">;</span>
<span class="o">...</span>
<span class="p">}</span>
</code></pre>
</div>
<h1 id="section-5">全局函数</h1>
<ul>
<li>Coro::nready</li>
</ul>
<p>返回在ready状态(即通过调用<code class="highlighter-rouge">schedule</code>可以切换的)的coro线程个数。值为<code class="highlighter-rouge">0</code>的话,就意味着唯一可运行的就是当前运行的这个线程。所以<code class="highlighter-rouge">cede</code>是没效果的,而<code class="highlighter-rouge">schedule</code>会死锁到有哪个空闲函数激活别的coro。</p>
<ul>
<li>my $guard = Coro::guard { … }</li>
</ul>
<p>这个函数还存在,不过早晚被废弃,请使用<code class="highlighter-rouge">Guard::guard</code>函数。</p>
<ul>
<li>unblock_sub { … }</li>
</ul>
<p>这个有用的工具接收一个块或者代码引用,然后“unblock”它,并返回一个新的代码引用。unblock意思是:调用新的代码引用会立刻返回,不阻塞,无返回值。而原本的代码会被另一个新的coro调用。</p>
<p>这个函数存在的原因是:很多event库(比如<a href="http://search.cpan.org/perldoc?Event">Event</a>库)是非线程安全(比较弱格式的可重入性)的。这意味着你在回调总不可以阻塞。否则你就可能收到崩溃的报警。我目前唯一知道可以不用<code class="highlighter-rouge">unblock_sub</code>就安全的event库就是<a href="http://search.cpan.org/perldoc?EV">EV</a>了(但是当你所有的事件循环都被block后,你还是会进入死锁状态)。</p>
<p>Coro会尝试在你在事件循环中被阻塞的时候捕获异常(FATAL:$Coro::IDLE blocked itself)。当然这只是近乎完美,而且还要求你不能用自己的循环实现。</p>
<p>这个函数允许你的回调是阻塞的,因为他会在另一个可以被安全阻塞的coro里执行。一个很常见的例子就是当你用<a href="http://search.cpan.org/perldoc?Coro::AIO">Coro::AIO</a>模块时,函数让你刷结果到磁盘上。</p>
<p>简单的说:在有阻塞可能的函数里用<code class="highlighter-rouge">unblock_sub</code>代替<code class="highlighter-rouge">sub</code>。</p>
<p>如果你的函数无所谓阻塞(比如给另一个coro发个信息,或者把其他coro整理到ready队列里),那就没理由用<code class="highlighter-rouge">unblock_sub</code>了。</p>
<p>注意你必须给C级别的事件循环中使用的回调函数使用<code class="highlighter-rouge">unblock_sub</code>。比如,当你使用一些用了<a href="http://search.cpan.org/perldoc?AnyEvent">AnyEvent</a>(而且你用的是<a href="http://search.cpan.org/perldoc?Coro::AnyEvent">Coro::AnyEvent</a>)的模块,这些模块提供的回调函数又是另一些事件回调的结果,你可不能阻塞掉它们,那么用<code class="highlighter-rouge">unblock_sub</code>吧。</p>
<ul>
<li>$cb = rouse_cb</li>
</ul>
<p>创建并返回一个“唤醒式的回调”。这是一个代码引用,当被调用的时候,它就记下调用的参数副本,然后通知拥有这个回调的coro。</p>
<ul>
<li>@args = rouse_wait [$cb]</li>
</ul>
<p>等待特定的唤醒回调(或者是本coro中最后创建的那个)。</p>
<p>一旦被调用(或者在<code class="highlighter-rouge">rouse_wait</code>之前被调用),他将返回最初传递给唤醒回调的参数。在标量上下文中意味着是<code class="highlighter-rouge">最后</code>一个参数,就好比<code class="highlighter-rouge">rouse_wait</code>最后状态是<code class="highlighter-rouge">return ($a1,$a2,$a3...)</code>。</p>
<p>参见下面__怎么等待回调__章节的实际使用例子。</p>
<h1 id="section-6">怎么等待回调</h1>
<p>对于一个coro线程,等待回调是非常常见的。当你在另一个事件驱动程序或者事件驱动库里使用coro的时候,很自然的触发它。</p>
<p>通常时注册一个回调函数对应相应的事件,然后当这个事件触发的时候调用这些函数。不过,你可能只是想等待事件,简单到极致了。</p>
<p>比如<code class="highlighter-rouge">AnyEvent->child</code>注册了一个回调到特定子进程退出的时候:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">my</span> <span class="nv">$child_watcher</span> <span class="o">=</span> <span class="nv">AnyEvent</span><span class="o">-></span><span class="nv">child</span> <span class="p">(</span><span class="nv">pid</span> <span class="o">=></span> <span class="nv">$pid</span><span class="p">,</span> <span class="nv">cb</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="o">...</span> <span class="p">});</span>
</code></pre>
</div>
<p>不过在coro里,你通常只需要这么写:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">my</span> <span class="nv">$status</span> <span class="o">=</span> <span class="nv">wait_for_child</span> <span class="nv">$pid</span><span class="p">;</span>
</code></pre>
</div>
<p>Coro提供了两个特定的函数让这件事情变得很容易:C<coro::rouse_cb>和C<coro::rouse_wait>。</coro::rouse_wait></coro::rouse_cb></p>
<p>第一个函数,C<rouse_cb>,生成并返回一个回调,当这个回调被调用时,会自动保存参数并通知创建该回调的coro。</rouse_cb></p>
<p>第二个函数,C<rouse_wait>,等待回调被调用(通过C<schedule>命令进入休眠)并返回传递给回调的初始参数。
使用这两个函数,就可以很容易的实现上面说的C<wait_for_child>函数了:</wait_for_child></schedule></rouse_wait></p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">sub </span><span class="nf">wait_for_child</span><span class="p">($)</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$pid</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$watcher</span> <span class="o">=</span> <span class="nv">AnyEvent</span><span class="o">-></span><span class="nv">child</span> <span class="p">(</span><span class="nv">pid</span> <span class="o">=></span> <span class="nv">$pid</span><span class="p">,</span> <span class="nv">cb</span> <span class="o">=></span> <span class="nn">Coro::</span><span class="nv">rouse_cb</span><span class="p">);</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$rpid</span><span class="p">,</span> <span class="nv">$rstatus</span><span class="p">)</span> <span class="o">=</span> <span class="nn">Coro::</span><span class="nv">rouse_wait</span><span class="p">;</span>
<span class="nv">$rstatus</span>
<span class="p">}</span>
</code></pre>
</div>
<p>如果嫌C<rouse_cb>和C<rouse_wait>还不够灵活,你还可以用C<schedule>自己搞起:</schedule></rouse_wait></rouse_cb></p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">sub </span><span class="nf">wait_for_child</span><span class="p">($)</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$pid</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="c1"># 把当前的coro存入$current,</span>
<span class="c1"># 然后提供一个结果变量传递给->child的闭包</span>
<span class="k">my</span> <span class="nv">$current</span> <span class="o">=</span> <span class="nv">$</span><span class="nn">Coro::</span><span class="nv">current</span><span class="p">;</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$done</span><span class="p">,</span> <span class="nv">$rstatus</span><span class="p">);</span>
<span class="c1"># pass a closure to ->child</span>
<span class="k">my</span> <span class="nv">$watcher</span> <span class="o">=</span> <span class="nv">AnyEvent</span><span class="o">-></span><span class="nv">child</span> <span class="p">(</span><span class="nv">pid</span> <span class="o">=></span> <span class="nv">$pid</span><span class="p">,</span> <span class="nv">cb</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">$rstatus</span> <span class="o">=</span> <span class="nv">$_</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span> <span class="c1"># 记住$rstatus</span>
<span class="nv">$done</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="c1"># 标记$rstatus</span>
<span class="p">});</span>
<span class="c1">#等待闭包被调用</span>
<span class="nv">schedule</span> <span class="k">while</span> <span class="o">!</span><span class="nv">$done</span><span class="p">;</span>
<span class="nv">$rstatus</span>
<span class="p">}</span>
</code></pre>
</div>
<h1 id="section-7">错误和限制</h1>
<ul>
<li>后端用pthread派生</li>
</ul>
<p>当coro使用pthread后端编译(不建议,但在一些BSD平台上不得不用,因为BSD的libcs完全不可用)的时候,coro无法生成fork。解决办法:修复glibc然后用snner后端。</p>
<ul>
<li>每个进程的仿真(线程)</li>
</ul>
<p>这个模块不是perl的伪线程安全。所以你只能在第一个线程里使用coro(未来的版本可能去掉这个要求,实现每个线程自己的schedule,不过当前Coro::State模块还不支持)。我建议关闭线程支持使用进程。因为开启windows进程仿真后,插入速度只有perl代码的一半。</p>
<p>注意,使用另一个进程创建出来的线程会崩溃(报错是空指针)。</p>
<ul>
<li>coro切换不是信号安全的</li>
</ul>
<p>你千万不要从一个处理sighal句柄的进程(指的是%SIG,大多数事件库都提供安全的信号)里切换到其他coro线程。_除非_你确信自己的做法不会中断Coro函数!</p>
<p>也就是说,你_绝对不_能调用任何可能阻塞当前coro的函数 —— <code class="highlighter-rouge">cede</code>,<code class="highlighter-rouge">schedule</code>,<code class="highlighter-rouge">Coro::Semaphore->down</code>或者其他使用了这些的函数。其他的命令,比如<code class="highlighter-rouge">ready</code>,则没问题。</p>
<h1 id="windows">windows进程仿真</h1>
<p>太多人看起来都对ithreads比较困惑(比如Chip Salzenberg就说我“无知,无能,愚蠢,上当了!”,而同一封邮件里他对perl的ithreads也是各种模糊的说法(比如说文件或者内存必须共享),这说明他在这方面了解甚微——如果对Chip来说这都很难理解,估计对所有人都没那么容易搞明白的)。</p>
<p>下面贴一段我在2009年perl聚会上分享的《脚本语言中的线程》的超浓缩版:</p>
<p>所谓的“ithreads”最初是为了这两个理由才实现的:第一,在原生win32平台的perl上模拟unix进程;第二,替代旧的,真正的线程模型(“5.005-threads”)。</p>
<p>最后的实现用线程替代了操作系统进程。进程和线程的区别是:同一个进程内的线程间是共享内存的(以及其他状态,比如文件)。而进程间可不会共享任何东西(至少语义上不会)。也就是说一个线程做的修改可以是其他线程可见的,而进程的修改是其他进程不可见的。</p>
<p>“ithreads”就是这样工作的:创建新的ithreads进程时,所有状态都被复制(内存是物理上实际复制,文件和代码是逻辑上的复制)。然后,所有修改被隔离。在UNIX上,这个行为是通过操作系统进程实现的。不过UNIX通常会使用构建进系统的硬件来有效的做到这点。而windows进程仿真是通过软件模拟这个硬件操作(也很有效,不过当然还是比硬件慢很多)。</p>
<p>所以,如上面说过的,加载代码,修改代码,修改数据结构,都只是所属ithreads内部可见。同一个OS进程内的其他ithreads是看不到的。</p>
<p>这就是为什么“ithreads”根本没有给perl实现线程,而依然是进程的原因。在非windows平台上,它表现相当糟糕,就是因为你完全可以利用硬件定制的优势(比如fork模块,它可以给你(i-)threads的API,而且快很多)。</p>
<p>要在ithreads模型里共享数据,只能在线程间通过缓慢的复制语义来传输数据结构——共享数据是不存在的。</p>
<p>i-threads交互密集型的基准测试显示结果相当糟糕(事实上糟糕到了没法直接利用多核优势的Coro都比它快上数量级。因为Coro可以在线程间共享数据,详见我的分享)。</p>
<p>综上所述,i-threads是用线程来实现了进程,也就是用fork的进程来模拟,嗯,进程。启用i-threads完全是拖累perl程序的运行,在非windows平台下完全没有(顶多算微乎其微的有)实用性,反而是损害那些单线程的perl程序。</p>
<p>这就是我避免用”ithreads”这个名字的原因,因为这完全是误导,听起来就跟它为perl实现了某种线程模型似的。我更喜欢的是“windows进程模拟”这个名字,这才更准确和真实的描述了它的实际作用和行为。</p>
<h1 id="section-8">另见</h1>
<p>事件循环集合: <a href="http://search.cpan.org/perldoc?Coro::AnyEvent">Coro::AnyEvent</a>,<a href="http://search.cpan.org/perldoc?Coro::EV">Coro::EV</a>,<a href="http://search.cpan.org/perldoc?Coro::Event">Coro::Event</a>。</p>
<p>调试: <a href="http://search.cpan.org/perldoc?Coro::Debug">Coro::Debug</a>。</p>
<p>支持/实用工具: <a href="http://search.cpan.org/perldoc?Coro::Specific">Coro::Specific</a>,<a href="http://search.cpan.org/perldoc?Coro::Util">Coro::Util</a>。</p>
<p>锁和过程间通信: <a href="http://search.cpan.org/perldoc?Coro::Signal">Coro::Signal</a>,<a href="http://search.cpan.org/perldoc?Coro::Channel">Coro::Channel</a>,<a href="http://search.cpan.org/perldoc?Coro::Semaphore">Coro::Semaphore</a>,<coro::semaphoreset>,[Coro::RWLock](http://search.cpan.org/perldoc?Coro::RWLock)。</coro::semaphoreset></p>
<p>I/O和定时器: <a href="http://search.cpan.org/perldoc?Coro::Timer">Coro::Timer</a>,<a href="http://search.cpan.org/perldoc?Coro::Handle">Coro::Handle</a>,<a href="http://search.cpan.org/perldoc?Coro::Socket">Coro::Socket</a>,<a href="http://search.cpan.org/perldoc?Coro::AIO">Coro::AIO</a>。</p>
<p>和其他模块的结合: <a href="http://search.cpan.org/perldoc?Coro::LWP">Coro::LWP</a>(不过实用的话建议选择<a href="http://search.cpan.org/perldoc?AnyEvent::HTTP">AnyEvent::HTTP</a>),<a href="http://search.cpan.org/perldoc?Coro::BDB">Coro::BDB</a>,<a href="http://search.cpan.org/perldoc?Coro::Storable">Coro::Storable</a>,<a href="http://search.cpan.org/perldoc?Coro::Select">Coro::Select</a>。</p>
<p>XS API: <a href="http://search.cpan.org/perldoc?Coro::MakeMaker">Coro::MakeMaker</a>。</p>
<p>底层配置,线程环境及延续机制: <a href="http://search.cpan.org/perldoc?Coro::State">Coro::State</a>。</p>
<h1 id="section-9">作者</h1>
<p>Marc Lehmann <a href="mailto:schmorp@schmorp.de">schmorp@schmorp.de</a><br />
http://home.schmorp.de/</p>
Chrome的APP简单用法
2012-11-09T00:00:00+08:00
web
chrome
http://chenlinux.com/2012/11/09/chrome-app-demo
<p>学习一下简单的chrome app写法。首先,chrome的ext和web app和packaged app就要分清楚。简单说,ext就是可以出现在地址栏右侧的,app是可以出现在任务栏右侧的。而web app其实就是用json描述了一个url地址,packaged app则是最接近普通桌面程序的,需要完整的带有html/css/js等内容。但同时,因为packaged app可以在关闭chrome浏览器后运行,所以有些浏览器上的API它也用不了。</p>
<p>首先上一个web app的demo,只要一个manifest.json在本地就够了。关键点是app里指定为url,permissions指定需要的权限。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span>
<span class="s2">"name"</span><span class="err">:</span> <span class="s2">"WebApp"</span><span class="p">,</span>
<span class="s2">"version"</span><span class="err">:</span> <span class="s2">"0.1"</span><span class="p">,</span>
<span class="s2">"app"</span><span class="err">:</span> <span class="p">{</span>
<span class="s2">"urls"</span><span class="err">:</span> <span class="p">[</span> <span class="s2">"http://www.domain.com/chrome/"</span> <span class="p">],</span>
<span class="s2">"launch"</span><span class="err">:</span> <span class="p">{</span>
<span class="s2">"web_url"</span><span class="err">:</span> <span class="s2">"http://www.domain.com/chrome/index.html"</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="s2">"permissions"</span><span class="err">:</span> <span class="p">[</span><span class="s2">"background"</span><span class="p">,</span> <span class="s2">"notifications"</span><span class="p">],</span>
<span class="s2">"manifest_version"</span><span class="err">:</span> <span class="mi">2</span>
<span class="p">}</span>
</code></pre>
</div>
<p>然后余下的事情就是web上的了。在<code class="highlighter-rouge">http://www.domain.com/chrome/index.html</code>里定义内容。比如我这是这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nt"><html><head></head></span>
<span class="nt"><body></span>
<span class="nt"><button</span> <span class="na">id=</span><span class="s">"openBackgroundWindow"</span><span class="nt">></span>开启后台运行<span class="nt"></button></span>
<span class="nt"><button</span> <span class="na">id=</span><span class="s">"closeBackgroundWindow"</span><span class="nt">></span>关闭后台运行<span class="nt"></button></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"index.js"</span><span class="nt">></script></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</code></pre>
</div>
<p>然后把事情交给index.js来完成。这也是chrome app的通常做法,尽量拆分干净,尤其到packaged app的时候,压根就不让你在html里写script了。index.js如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="kd">var</span> <span class="nx">bgWinUrl</span> <span class="o">=</span> <span class="s2">"background.html#yay"</span><span class="p">;</span>
<span class="kd">var</span> <span class="nx">bgWinName</span> <span class="o">=</span> <span class="s2">"bgNotifier"</span><span class="p">;</span>
<span class="kd">function</span> <span class="nx">openBackgroundWindow</span><span class="p">()</span> <span class="p">{</span>
<span class="nb">window</span><span class="p">.</span><span class="nx">open</span><span class="p">(</span><span class="nx">bgWinUrl</span><span class="p">,</span> <span class="nx">bgWinName</span><span class="p">,</span> <span class="s2">"background"</span><span class="p">);</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">closeBackgroundWindow</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">w</span> <span class="o">=</span> <span class="nb">window</span><span class="p">.</span><span class="nx">open</span><span class="p">(</span><span class="nx">bgWinUrl</span><span class="p">,</span> <span class="nx">bgWinName</span><span class="p">,</span> <span class="s2">"background"</span><span class="p">);</span>
<span class="nx">w</span><span class="p">.</span><span class="nx">close</span><span class="p">();</span>
<span class="p">}</span>
<span class="nb">document</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="s1">'DOMContentLoaded'</span><span class="p">,</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="nb">document</span><span class="p">.</span><span class="nx">querySelector</span><span class="p">(</span><span class="s1">'#openBackgroundWindow'</span><span class="p">).</span><span class="nx">addEventListener</span><span class="p">(</span>
<span class="s1">'click'</span><span class="p">,</span> <span class="nx">openBackgroundWindow</span><span class="p">);</span>
<span class="nb">document</span><span class="p">.</span><span class="nx">querySelector</span><span class="p">(</span><span class="s1">'#closeBackgroundWindow'</span><span class="p">).</span><span class="nx">addEventListener</span><span class="p">(</span>
<span class="s1">'click'</span><span class="p">,</span> <span class="nx">closeBackgroundWindow</span><span class="p">);</span>
<span class="p">});</span>
</code></pre>
</div>
<p>注意到这里的<code class="highlighter-rouge">background.html</code>加了锚点,原因我懒得看英文说明了,反正大意是不写个锚点有时候会出问题。</p>
<p>最后是background.html,内容参见之前博客里写的juggernaut。简单几十行。一个可以不再打开具体页面自动收报警消息的app就改造出来了~~</p>
<p>这时候进一步折腾的想法就出来了:既然叫app嘛,功能应该多一点,不单收报警,还能存下来,这样出去一趟回来可以看离线记录。</p>
<p>下面就说离线的packaged app。</p>
<p><code class="highlighter-rouge">manifest.json</code>的app里就不能写urls要写background了。文档中说background可以写scripts或者page,但是我实验发现scripts正常page不起作用(也确实不报错)。<br />
<code class="highlighter-rouge">javascript
{
"name": "Packaged App",
"version": "0.1",
"app": {
"background": {
"scripts": ["juggernaut.js", "filesystem.js", "background.js"]
}
},
"permissions": ["background", "notifications", "unlimitedStorage"],
"manifest_version": 2
}
</code><br />
chrome app会自动把scripts数组<code class="highlighter-rouge">依次</code>加载。<br />
这里需要注意修改一下默认的juggernaut/application.js,因为chrome packaged app没有浏览器框架,所以disconnect那有个报错,删除掉那三行即可。<br />
然后background.js里最常见的功能是当点击app的时候弹出的页面,代码如下:<br />
<code class="highlighter-rouge">javascript
chrome.app.runtime.onLaunched.addListener(function() {
chrome.app.window.create('main.html', {
'width': 400,
'height': 500,
//frame: 'none'
});
});
</code><br />
如果css够好,可以开启<code class="highlighter-rouge">frame: none</code>,然后页面看不到一丝浏览器的样子,你就可以做得跟真的app一样有自己的控制了。</p>
<p>然后是文件操作。html5有file api。所以可以直接操作文件了:<br />
```javascript<br />
window.webkitRequestFileSystem(window.TEMPORARY, 5<em>1024</em>1024, function (fs) {<br />
fs.root.getFile(“syslog.txt”, {create: true}, function(fileEntry){<br />
fileEntry.createWriter(function(fileWriter){<br />
var bb = new Blob([“New File Ready\n”], {type: ‘text/plain’});<br />
fileWriter.write(bb);<br />
}, errorHandler);<br />
}, errorHandler);<br />
}, errorHandler);</p>
<p>function append_file(msg) {<br />
window.webkitRequestFileSystem(window.TEMPORARY, 5<em>1024</em>1024, function (fs) {<br />
fs.root.getFile(“syslog.txt”, {create: false}, function(fileEntry){<br />
fileEntry.createWriter(function(fileWriter){<br />
fileWriter.seek(fileWriter.length);<br />
var bb = new Blob([msg], {type: ‘text/plain’});<br />
fileWriter.write(bb);<br />
}, errorHandler);<br />
}, errorHandler);<br />
}, errorHandler);<br />
};<br />
```<br />
这里大多数网上的例子都还是用的<code class="highlighter-rouge">BlobBuilder()</code>,经过我试验,至少在chromium version24上,已经没有这个API了。</p>
<p>这里注意一个问题:Juggernaut同时收到多条消息,调用append_file()的话,会有文件锁问题。所以我加上一个缓冲控制:<br />
<code class="highlighter-rouge">javascript
var message = '';
setInterval(function() {
if ( message ) {
append_file(message);
message = '';
}
}, 60000);
jug.subscribe("syslog", function(data){
var msg = data.join('|') + "\n";
message += msg;
...
notification.show();
});
</code><br />
后台的工作大概就是这些。然后是弹出的main.html。记住之前提到的,不能在里面写js。所以html里除了写个div啥都没有,功能依然交给main.js来做:<br />
```html</p>
<html>
<head>
<meta charset="utf-8" />
<title>Storage test</title>
<script src="main.js"></script>
</head>
<body>
<div id="bar">
<form id="form">
<button id="reload-window-button">刷新</button>
<button id="remove-window-button">清空</button>
<button id="close-window-button">关闭</button>
</form>
</div>
<div id="log">
<ul id="msg">
</ul>
</div>
</body>
</html>
<div class="highlighter-rouge"><pre class="highlight"><code>不过关于刷新页面的问题现在还比较茫然,试过metadata/reload/location.href等各种办法,都不起作用,只能在页面上右键选择刷新……
```javascript
onload = function() {
var channel_file = 'syslog.txt';
window.webkitRequestFileSystem(window.TEMPORARY, 5*1024*1024, function (fs) {
fs.root.getFile(channel_file, {}, function(fileEntry) {
fileEntry.file(function(file) {
var reader = new FileReader();
reader.onloadend = function() {
var logul = document.getElementById('log');
var logli = document.createElement('li');
logli.innerText = this.result;
logul.insertBefore(logli, logul.firstChild);
};
reader.readAsText(file);
}, errorHandler);
}, errorHandler);
}, errorHandler);
document.getElementById("remove-window-button").onclick = function() {
window.webkitRequestFileSystem(window.TEMPORARY, 5*1024*1024, function (fs) {
fs.root.getFile(channel_file, {create: false}, function(fileEntry){
fileEntry.remove(function(){
fs.root.getFile(channel_file, {create: true}, function(fileEntry){
fileEntry.createWriter(function(fileWriter){
var bb = new Blob(["Clean Over\n"], {type: 'text/plain'});
fileWriter.write(bb);
}, errorHandler);
}, errorHandler);
}, errorHandler);
}, errorHandler);
}, errorHandler);
}
document.getElementById("close-window-button").onclick = function() {
window.close();
}
document.getElementById("reload-window-button").onclick = function() {
window.location.reload(true);
}
};
</code></pre>
</div>
<p>以上。</p>
<p>最后,如果完成了,在浏览器上直接选择打包即可。不过新版本的chrome已经不再允许直接安装非web store的crx了……</p>
【Logstash系列】Outputs::ElasticsearchHTTP自动获取随机node
2012-10-30T00:00:00+08:00
logstash
ruby
elasticsearch
http://chenlinux.com/2012/10/30/faraday-select-rand-node-for-logstash-output-elasticsearch-http
<p>今天在ES群中和medcl请教了一下index的性能问题。基本上在bulk的基础上,还有几点是可以做的。当然medcl说的是正常的全文索引的场景:</p>
<ul>
<li>不用http协议,直接走tcp层,维护一个pool发bulk;</li>
<li>多台node的情况下,在bulk前先设置replica为0;bulk完成后再调整replica;</li>
<li>因为es会自动路由,所以index请求可以分散开直接发给多个node。</li>
</ul>
<p>总之,就是减少集群内部的网络传输。</p>
<p>介于logstash的应用是一直持续往es写数据的,所以replica调整这招用不上,顶多加大refresh时间而已。所以可以动手的地方主要就是第三条了。</p>
<p>正好去翻了一下perl的Elasticsearch.pm的POD。发现原来perl模块本色默认就是这么做的。new的时候定义的server,是用来发送请求获取集群所有alive的nodes。然后会从这个nodes列表里选择(随机)一个创建真正的链接返回。获取nodes的API如下:<br />
<code class="highlighter-rouge">bash
curl http://192.168.1.33:9200/_cluster/nodes?pretty=1
{
"ok" : true,
"cluster_name" : "logstash",
"nodes" : {
"he6ipuA3SDeNmOQYIr-bjg" : {
"name" : "Afari, Jamal",
"transport_address" : "inet[/192.168.1.33:9300]",
"hostname" : "ES-33.domain.com",
"http_address" : "inet[/192.168.1.33:9200]"
},
"WXK68VX0ThmNnozq0uioQw" : {
"name" : "Harker, Quincy",
"transport_address" : "inet[/192.168.1.68:9300]",
"hostname" : "ES-68.domain.com",
"http_address" : "inet[/192.168.1.68:9200]"
}
}
}
</code><br />
这样显然可以在cluster较大的时候分担index的压力(search的时候压力在集群本身的cpu上)。我打算给我的pure-ruby branch里的faraday版的Logstash::Outputs::ElasticsearchHTTP也加上这个功能。</p>
<hr />
<p>大致简单实现如下:<br />
<code class="highlighter-rouge">ruby
def select_rand_host
require "json"
begin
response = @agent.get '/_cluster/nodes'
nodelist = JSON.parse(response.body)['nodes'].values
livenode = nodelist[rand(nodelist.length)]["http_address"].split(/\[|\]/)[1]
@agent = Faraday.new(:url => "http:/#{livenode}") do |faraday|
faraday.use Faraday::Adapter::EMHttp
end
end
end
</code><br />
然后在<code class="highlighter-rouge">def register</code>里和<code class="highlighter-rouge">def flush</code>的<code class="highlighter-rouge">retry</code>前面都加上<code class="highlighter-rouge">select_rand_host()</code>就好了。当然比起perl的ElasticSearch::Transport里各种检查各种排除,我这个还是简单多了……<br />
另,在Ruby1.9里从数组返回随机元素可以直接调用<code class="highlighter-rouge">.sample</code>,真赞。不过谁让我都是1.8.7的版本呢……</p>
<p><strong>2012 年 12 月 30 日附注:</strong></p>
<p>之后我在 maillist 里联系了 logstash 的作者。不过作者表示:第一,需要自选 node 功能的,建议使用 output/elasticsearch 因为这个是用的 java 客户端,直接会把自己作为一个 node 加入 ES 的 cluster。第二,ruby 的 http 模块他都不喜欢,所以全部项目里的相关部分他都只用自己写的 ftw 模块 ==!</p>
<p><strong>2013 年 02 月 26 日附注:</strong></p>
<p>最新版的 Logstash::Outputs::ElasticSearchHTTP 已经加入随机选择 node 功能。是其他网友在 ftw 模块基础上添加的。</p>
用systemtap调试文件描述符限制
2012-10-26T00:00:00+08:00
linux
systemtap
C
http://chenlinux.com/2012/10/26/systemtap-and-file-descriptor
<p>在运行一些非root用户进程的时候,我们都习惯要在前面加上一个ulimit -HSn 65535的命令。而且我们还知道关于文件描述符的限制,不止这一个地方,还有limits.conf,sysctl -w fs.file-max等等。但是到底这些是什么个关系呢?而且,如果是一个已经在运行的程序,有没有可能在更改他的文件描述符限制呢?</p>
<p>以最经常碰到这种情况的squid为例。我们可以在squid/src/comm.c里看到<code class="highlighter-rouge">comm_openex()</code>是如何发现超出限制的,嗯,可以说没有”自己发现”,直接判断socket()是否成功而已。所以接下来的事情是看socket创建过程怎么判断的。</p>
<p>关于socket过程,主要看kernel/net/socket.c和kernel/fs/file.c,作为C菜鸟,如下系列博文在这方面说的非常清楚,我就不详细说了:</p>
<p><a href="http://blog.chinaunix.net/uid-23629988-id-3080166.html">TCP/IP源码学习(47)——socket与VFS的关联(1)</a></p>
<p><a href="http://blog.chinaunix.net/uid-23629988-id-3083376.html">TCP/IP源码学习(48)——socket与VFS的关联(2)</a></p>
<p>大体来说,就是socket本身是不触及这个限制的,但是创建出来的socket必须和文件描述符关联起来(<code class="highlighter-rouge">sock_map_fd()</code> – <code class="highlighter-rouge">sock_alloc_file()</code> – <code class="highlighter-rouge">get_unused_fd_flags()</code> – <code class="highlighter-rouge">alloc_fd()</code>),这就相关了。在<code class="highlighter-rouge">alloc_fd()</code>中会读取当前进程(current)的fdtable,fdtable的结构由”linux/fdtable.h”定义如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">struct</span> <span class="n">fdtable</span> <span class="p">{</span>
<span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">max_fds</span><span class="p">;</span>
<span class="k">struct</span> <span class="n">file</span> <span class="o">**</span> <span class="n">fd</span><span class="p">;</span> <span class="cm">/* current fd array */</span>
<span class="n">fd_set</span> <span class="o">*</span><span class="n">close_on_exec</span><span class="p">;</span>
<span class="n">fd_set</span> <span class="o">*</span><span class="n">open_fds</span><span class="p">;</span>
<span class="k">struct</span> <span class="n">rcu_head</span> <span class="n">rcu</span><span class="p">;</span>
<span class="k">struct</span> <span class="n">fdtable</span> <span class="o">*</span><span class="n">next</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">struct</span> <span class="n">files_struct</span> <span class="p">{</span>
<span class="n">atomic_t</span> <span class="n">count</span><span class="p">;</span>
<span class="k">struct</span> <span class="n">fdtable</span> <span class="o">*</span><span class="n">fdt</span><span class="p">;</span>
<span class="k">struct</span> <span class="n">fdtable</span> <span class="n">fdtab</span><span class="p">;</span>
<span class="n">spinlock_t</span> <span class="n">file_lock</span> <span class="n">____cacheline_aligned_in_smp</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">next_fd</span><span class="p">;</span>
<span class="k">struct</span> <span class="n">embedded_fd_set</span> <span class="n">close_on_exec_init</span><span class="p">;</span>
<span class="k">struct</span> <span class="n">embedded_fd_set</span> <span class="n">open_fds_init</span><span class="p">;</span>
<span class="k">struct</span> <span class="n">file</span> <span class="o">*</span> <span class="n">fd_array</span><span class="p">[</span><span class="n">NR_OPEN_DEFAULT</span><span class="p">];</span>
<span class="p">};</span>
<span class="cp">#define files_fdtable(files) (rcu_dereference((files)->fdt))
</span></code></pre>
</div>
<p>打开fd的过程如下:</p>
<ol>
<li>如果有files->next_fd,直接使用;</li>
<li>否则从fdtable->open_fds->fds_bits[]找到fdtable->max_fds,到找到一个可用的为止;</li>
<li>否则说明当前fdtable不够用,需要扩充expand_files();</li>
<li>完成后把获得的fd+1赋值给files->next_fd,这样下次就可以直接用;</li>
<li>最后把这个fd加入fdtable->open_fds里。</li>
</ol>
<p>那么涉及max open files的显然就是这个<code class="highlighter-rouge">expand_files()</code>了。继续看,可以发现其中的判断分三部分:</p>
<ol>
<li>if (nr >= current->signal->rlim[RLIMIT_NOFILE].rlim_cur)</li>
<li>if (nr < fdt->max_fds)</li>
<li>if (nr >= sysctl_nr_open)</li>
</ol>
<p>都逃过之后才进入<code class="highlighter-rouge">expand_fdtable()</code>真正扩展。很好,现在我们看到之前就知道的ulimit/sysctl神马的是怎么限定的了。那么这第二个呢?我们可以找到files这个结构的init,如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">struct</span> <span class="n">files_struct</span> <span class="n">init_files</span> <span class="o">=</span> <span class="p">{</span>
<span class="p">.</span><span class="n">count</span> <span class="o">=</span> <span class="n">ATOMIC_INIT</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span>
<span class="p">.</span><span class="n">fdt</span> <span class="o">=</span> <span class="o">&</span><span class="n">init_files</span><span class="p">.</span><span class="n">fdtab</span><span class="p">,</span>
<span class="p">.</span><span class="n">fdtab</span> <span class="o">=</span> <span class="p">{</span>
<span class="p">.</span><span class="n">max_fds</span> <span class="o">=</span> <span class="n">NR_OPEN_DEFAULT</span><span class="p">,</span>
<span class="p">.</span><span class="n">fd</span> <span class="o">=</span> <span class="o">&</span><span class="n">init_files</span><span class="p">.</span><span class="n">fd_array</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span>
<span class="p">.</span><span class="n">close_on_exec</span> <span class="o">=</span> <span class="p">(</span><span class="n">fd_set</span> <span class="o">*</span><span class="p">)</span><span class="o">&</span><span class="n">init_files</span><span class="p">.</span><span class="n">close_on_exec_init</span><span class="p">,</span>
<span class="p">.</span><span class="n">open_fds</span> <span class="o">=</span> <span class="p">(</span><span class="n">fd_set</span> <span class="o">*</span><span class="p">)</span><span class="o">&</span><span class="n">init_files</span><span class="p">.</span><span class="n">open_fds_init</span><span class="p">,</span>
<span class="p">.</span><span class="n">rcu</span> <span class="o">=</span> <span class="n">RCU_HEAD_INIT</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">.</span><span class="n">file_lock</span> <span class="o">=</span> <span class="n">__SPIN_LOCK_UNLOCKED</span><span class="p">(</span><span class="n">init_task</span><span class="p">.</span><span class="n">file_lock</span><span class="p">),</span>
<span class="p">};</span>
</code></pre>
</div>
<p>这个<code class="highlighter-rouge">NR_OPEN_DEFAULT</code>可以在fdtable.h里看到就是<code class="highlighter-rouge">BITS_PER_LONG</code>。<code class="highlighter-rouge">BITS_PER_LONG</code>应该是32或者64,取决于CPU是32还是64位的了。</p>
<p>其实这里还可以继续看<code class="highlighter-rouge">alloc_fdtable()</code>中怎么确定新扩展的<code class="highlighter-rouge">fdt->max_fds</code>的,如果nr大于sysctl的设定,那么nr会计算成</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="p">((</span><span class="n">sysctl_nr_open</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">|</span> <span class="p">(</span><span class="n">BITS_PER_LONG</span> <span class="o">-</span> <span class="mi">1</span><span class="p">))</span> <span class="o">+</span> <span class="mi">1</span>
</code></pre>
</div>
<p>然后<code class="highlighter-rouge">copy_fdtable()</code>转移数据,奇怪的是看到转移完后,还判断了原有<code class="highlighter-rouge">fdt->max_fds > NR_OPEN_DEFAULT</code>才释放,我不清楚什么情况下会有<code class="highlighter-rouge">fdt->max_fds</code>小于init值了…</p>
<p>回到主题。三个条件里后两个条件都很明白了。现在就是第一个,这是根据current不同有不同的,我们可以试试看如果修改这个值会怎么样?</p>
<p>修改工具我用到了systemtap大神器,不过我是菜鸟啦~脚本如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="cp">#!/usr/bin/stap
</span> <span class="o">%</span><span class="p">{</span>
<span class="cp">#include <linux/sched.h>
</span> <span class="cp">#include <linux/resource.h>
</span> <span class="o">%</span><span class="p">}</span>
<span class="n">probe</span> <span class="n">begin</span> <span class="p">{</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"begin...</span><span class="se">\n</span><span class="s">"</span><span class="p">)</span>
<span class="p">}</span>
<span class="n">probe</span> <span class="n">kernel</span><span class="p">.</span><span class="n">function</span><span class="p">(</span><span class="s">"expand_files@fs/file.c"</span><span class="p">).</span><span class="n">call</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span> <span class="n">execname</span><span class="p">()</span> <span class="o">==</span> <span class="s">"squid"</span> <span class="p">)</span> <span class="p">{</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"[%s] %s fdt:%d, task:%d, rlim:%d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">tz_ctime</span><span class="p">(</span><span class="n">gettimeofday_s</span><span class="p">()),</span> <span class="n">execname</span><span class="p">(),</span> <span class="err">$</span><span class="n">files</span><span class="o">-></span><span class="n">fdtab</span><span class="o">-></span><span class="n">max_fds</span><span class="p">,</span> <span class="n">task_open_file_handles</span><span class="p">(</span><span class="n">task_current</span><span class="p">()),</span> <span class="n">rlim_cur</span><span class="p">());</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"</span><span class="se">\t</span><span class="s">args_nr: %d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="err">$</span><span class="n">nr</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span> <span class="n">rlim_cur</span><span class="p">()</span> <span class="o"><</span> <span class="err">$</span><span class="mi">1</span> <span class="p">)</span> <span class="p">{</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"</span><span class="se">\t</span><span class="s">set rlim: %d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">set_rlim_cur</span><span class="p">(</span><span class="err">$</span><span class="mi">1</span><span class="p">));</span>
<span class="n">exit</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">probe</span> <span class="n">kernel</span><span class="p">.</span><span class="n">function</span><span class="p">(</span><span class="s">"expand_files@fs/file.c"</span><span class="p">).</span><span class="k">return</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span> <span class="n">execname</span><span class="p">()</span> <span class="o">==</span> <span class="s">"squid"</span> <span class="p">)</span> <span class="p">{</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"</span><span class="se">\t</span><span class="s">return: %d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="err">$</span><span class="k">return</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">probe</span> <span class="n">kernel</span><span class="p">.</span><span class="n">function</span><span class="p">(</span><span class="s">"expand_fdtable"</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"%s call fdtable with %s"</span><span class="p">,</span> <span class="n">execname</span><span class="p">(),</span> <span class="err">$$</span><span class="n">vars</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">function</span> <span class="n">rlim_cur</span><span class="o">:</span><span class="kt">long</span> <span class="p">()</span>
<span class="o">%</span><span class="p">{</span> <span class="cm">/* pure */</span> <span class="cm">/* unprivileged */</span>
<span class="k">struct</span> <span class="n">signal_struct</span> <span class="o">*</span><span class="n">ss</span> <span class="o">=</span> <span class="n">kread</span><span class="p">(</span> <span class="o">&</span><span class="p">(</span><span class="n">current</span><span class="o">-></span><span class="n">signal</span><span class="p">)</span> <span class="p">);</span>
<span class="n">THIS</span><span class="o">-></span><span class="n">__retvalue</span> <span class="o">=</span> <span class="n">kread</span> <span class="p">(</span><span class="o">&</span><span class="p">(</span><span class="n">ss</span><span class="o">-></span><span class="n">rlim</span><span class="p">[</span><span class="n">RLIMIT_NOFILE</span><span class="p">].</span><span class="n">rlim_cur</span><span class="p">));</span>
<span class="n">CATCH_DEREF_FAULT</span><span class="p">();</span>
<span class="o">%</span><span class="p">}</span>
<span class="n">function</span> <span class="n">set_rlim_cur</span><span class="o">:</span><span class="kt">long</span> <span class="p">(</span><span class="n">val</span><span class="o">:</span><span class="kt">long</span><span class="p">)</span>
<span class="o">%</span><span class="p">{</span> <span class="cm">/* pure */</span> <span class="cm">/* unprivileged */</span>
<span class="k">struct</span> <span class="n">signal_struct</span> <span class="o">*</span><span class="n">ss</span> <span class="o">=</span> <span class="n">kread</span><span class="p">(</span> <span class="o">&</span><span class="p">(</span><span class="n">current</span><span class="o">-></span><span class="n">signal</span><span class="p">)</span> <span class="p">);</span>
<span class="n">kwrite</span><span class="p">(</span><span class="o">&</span><span class="p">(</span><span class="n">ss</span><span class="o">-></span><span class="n">rlim</span><span class="p">[</span><span class="n">RLIMIT_NOFILE</span><span class="p">].</span><span class="n">rlim_cur</span><span class="p">),</span> <span class="n">THIS</span><span class="o">-></span><span class="n">val</span><span class="p">);</span>
<span class="n">CATCH_DEREF_FAULT</span><span class="p">();</span>
<span class="o">%</span><span class="p">}</span>
</code></pre>
</div>
<p>systemtap自己提供了一系列tapset函数,比如这里的execname(),task_*都是。注意systemtap是脚本语言的,所以这些函数直接在/usr/share/systemtap/下面可以看怎么写的。比如我上面定义的两个function就是仿照里面<code class="highlighter-rouge">task_max_file_handles()</code>写的。</p>
<p>用%和{}标记的是内嵌C代码,systemtap在编译成C的时候直接插入进去,可以stap -k保留在/tmp/stap123456下查看的到。</p>
<p>kread/kwrite是systemtap-runtime提供的函数,封装的是put_user/get_user指令。</p>
<p>现在我们启动squid进程和stap脚本:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">ulimit</span> -HSn 256;squid -D
stap -g max_fds.stp 1024
</code></pre>
</div>
<p>注意要加-g,否则不会加载内嵌C的。<br />
另开窗口发起一次请求,然后看到stap输出:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>begin...
[Fri Oct 26 19:20:34 2012 CST] squid fdt:64, task:16, rlim:256
args_nr: 12
set rlim: 0
</code></pre>
</div>
<p>额,这个set是返回值0。function里如果把kwrite的返回值赋给<code class="highlighter-rouge">THIS->retvalue</code>会报void的错误。挺奇怪的。</p>
<p>再运行stap,发起请求,就可以看到squid的rlim变成1024了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>begin...
[Fri Oct 26 19:24:40 2012 CST] squid fdt:64, task:16, rlim:1024
args_nr: 12
return: 0
[Fri Oct 26 19:24:40 2012 CST] squid fdt:64, task:17, rlim:1024
args_nr: 17
return: 0
</code></pre>
</div>
<p>不过我的虚拟机跑不出这么大的并发,没法超过<code class="highlighter-rouge">BITS_PER_LONG</code>的64。所以我们可以换一个思路来验证<code class="highlighter-rouge">expand_files()</code>的执行。</p>
<ul>
<li>先ulimit -HSn 16启动squid,然后stap修改到1024</li>
</ul>
<p>这个时候,搞笑而现实的事情发生了。nr越过了1024的判断,却越不过<code class="highlighter-rouge">BITS_PER_LONG</code>的判断,于是<code class="highlighter-rouge">expand_files</code>永远过不去。squid的max open files只能停留在16。这一步的return是0。所以看到stap里.return{}打印的是0。</p>
<ul>
<li>先ulimit -HSn 1024启动squid,然后stap修改到16</li>
</ul>
<p>这个时候,由之前的测试可以看到,squid本身就要用掉十多个fd来维护运行的。所以基本一接请求nr就超过16了。squid完全无法响应请求。这一步的return是-EMFILE,于是屏幕上开始出现一行行squid call fdtable {.n}………</p>
<p>附注:使用systemtap需要debuginfo。内核调试需要kernel-debuginfo/kernel-devel,程序也需要。如果是自己编译的,没问题直接probe process(“${path}/command”)即可,如果是rpm安装的,那就必须得把debuginfo包安装上,比如nginx-debuginfo.rpm这样子。</p>
<p>2012年12月3日更新:</p>
<p>采用tc延时的办法,可以达到在虚拟机上获取squid高连接的模拟环境。然后作出如下修正:</p>
<ul>
<li>在修改<code class="highlighter-rouge">rlim_cur</code>的时候也需要修改<code class="highlighter-rouge">rlim_max</code></li>
</ul>
<p>在上面的缩小测试里不会触发问题,不过在增大的测试中,问题来了——<code class="highlighter-rouge">setrlimit()</code>函数会很诧异为毛自己参数里那个<code class="highlighter-rouge">&rl</code>的<code class="highlighter-rouge">rlim_cur</code>比<code class="highlighter-rouge">rlim_max</code>还大?然后悲剧的报出”etrlimit(RLIMIT_NOFILE) failed: Invalid argument (22)”的错误……</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">kwrite</span><span class="p">(</span><span class="o">&</span><span class="p">(</span><span class="n">ss</span><span class="o">-></span><span class="n">rlim</span><span class="p">[</span><span class="n">RLIMIT_NOFILE</span><span class="p">].</span><span class="n">rlim_max</span><span class="p">),</span> <span class="n">THIS</span><span class="o">-></span><span class="n">val</span><span class="p">);</span>
</code></pre>
</div>
<ul>
<li>对于squid还需要修改<code class="highlighter-rouge">Squid_MaxFD</code>全局变量</li>
</ul>
<p>上面都是对kernel里的socket和file的修改,在实际运用中,用户程序本身也会有各种判断。squid维护了一堆全局变量,比如<code class="highlighter-rouge">Squid_MaxFD</code>,<code class="highlighter-rouge">Biggest_FD</code>和<code class="highlighter-rouge">Number_FD</code>,这就是squidclient mgr:info里看到的关于文件描述符的那几个值。其中<code class="highlighter-rouge">Squid_MaxFD</code>是在init的时候根据主进程启动时的ulimit情况一次性设定的,即便child进程重启也不会变。而<code class="highlighter-rouge">Biggest_FD</code>则是由<code class="highlighter-rouge">fdUpdateBiggest()</code>函数每次更新。不巧的是,里面有这么一句判断:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="n">assert</span><span class="p">(</span><span class="n">fd</span> <span class="o"><</span> <span class="n">Squid_MaxFD</span><span class="p">);</span>
</code></pre>
</div>
<p>所以,光修改kernel里的限制,socket返回后在更新<code class="highlighter-rouge">Biggest_FD</code>时squid会直接挂掉……</p>
<p>下面是修改squid进程里全局变量的办法,和修改kernel其实很类似:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">probe</span> <span class="nf">process</span><span class="p">(</span><span class="s">"/usr/sbin/squid"</span><span class="p">).</span><span class="n">function</span><span class="p">(</span><span class="s">"fdUpdateBiggest@src/fd.c"</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span> <span class="err">$</span><span class="n">Squid_MaxFD</span> <span class="o"><</span> <span class="mi">65535</span> <span class="p">)</span> <span class="p">{</span>
<span class="err">$</span><span class="n">Squid_MaxFD</span> <span class="o">=</span> <span class="mi">65535</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>把上面两个修改加入到之前的文件,然后测试增大ulimit限制(记住ulimit要大于64,否则不起作用哟),就没问题了。</p>
【Logstash系列】ElasticSearch的几点使用事项
2012-10-21T00:00:00+08:00
logstash
elasticsearch
http://chenlinux.com/2012/10/21/elasticearch-simple-usage
<p>之前已经写过一些ES的使用,也翻译了一篇官网上关于ES存储日志的建议日志。今天稍微总结一下近期以来实践出来的方案。</p>
<h1 id="shardreplica">shard和replica的选择</h1>
<p>在测试期,可以单节点上设置成1 shard + 0 replica的方式,这种的indexing速度是最快的(存疑:我至今没搞清楚ES在index的时候集群”应该”是比单node快还是慢)。</p>
<p>我曾经按照”常理”(我想象中的)理解,设定成10 shards + 0 replica,期望能用上并行写双node加倍index,事实上压根没用,而且因为另一台node上有其他负载的原因导致更慢了。</p>
<p>更可怕的是:就在前几天,突然出现一个shard挂了,……毫无办法,整个index全作废了。</p>
<p>所以结论是:无论如何,一定要保证有 > 0 份的replica!! 至于shards,保持默认的5个,或者顶多到20个也就差不多了。在maillist里看到有哥们设了100个,然后苦着脸问性能问题…… 要知道shards的份数是一旦设定不能更改的。</p>
<h1 id="template">template的使用</h1>
<p>刚开始的时候,每次实验都去改/etc/elasticsearch/elasticsearch.yml配置文件。事实上在template里修改settings更方便而且灵活!当然最主要的,还是调节里面的properties设定,合理的控制store和analyze了。</p>
<p>template设定也有多种方法。最简单的就是和存储数据一样POST上去。长期的办法,就是写成json文件放在配置路径里。其中,default配置放在/etc/elasticsearch/下,其他配置放在/etc/elasticsearch/templates/下。举例我现在的一个templates/template-logstash.json内容如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nt">"template-logstash"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"template"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"logstash*"</span><span class="p">,</span><span class="w">
</span><span class="nt">"settings"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"index.number_of_shards"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="p">,</span><span class="w">
</span><span class="nt">"number_of_replicas"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w">
</span><span class="nt">"index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"store"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"compress"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"stored"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="nt">"tv"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"mappings"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_default_"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"dynamic"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"true"</span><span class="p">,</span><span class="w">
</span><span class="err">}</span><span class="p">,</span><span class="w">
</span><span class="err">}</span><span class="p">,</span><span class="w">
</span><span class="nt">"loadbalancer"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"_source"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"compress"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="err">}</span><span class="p">,</span><span class="w">
</span><span class="nt">"_ttl"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"enabled"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="nt">"default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"10d"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"_all"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"enabled"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"@fields"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"dynamic"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"true"</span><span class="p">,</span><span class="w">
</span><span class="nt">"properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"client"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="p">,</span><span class="w">
</span><span class="nt">"index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"not_analyzed"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"domain"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="p">,</span><span class="w">
</span><span class="nt">"index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"not_analyzed"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"oh"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="p">,</span><span class="w">
</span><span class="nt">"index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"not_analyzed"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"responsetime"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"double"</span><span class="p">,</span><span class="w">
</span><span class="err">}</span><span class="p">,</span><span class="w">
</span><span class="nt">"size"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"long"</span><span class="p">,</span><span class="w">
</span><span class="nt">"index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"not_analyzed"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"status"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="p">,</span><span class="w">
</span><span class="nt">"index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"not_analyzed"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"upstreamtime"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"double"</span><span class="p">,</span><span class="w">
</span><span class="err">}</span><span class="p">,</span><span class="w">
</span><span class="nt">"url"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="p">,</span><span class="w">
</span><span class="nt">"index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"not_analyzed"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"@source"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="p">,</span><span class="w">
</span><span class="nt">"index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"not_analyzed"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"@timestamp"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"date"</span><span class="p">,</span><span class="w">
</span><span class="nt">"format"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"dateOptionalTime"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"@type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="p">,</span><span class="w">
</span><span class="nt">"index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"not_analyzed"</span><span class="p">,</span><span class="w">
</span><span class="nt">"store"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"no"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
</div>
<p><strong>注意:POST 发送的 json 内容比存储的 json 文件内容要少最外层的名字,因为名字是在 url 里体现的。</strong></p>
<h1 id="mapping">mapping简介</h1>
<p>上面template中除了index/shard/replica之外的部分,就是mapping了,大家注意到其中的dynamic,默认情况下,index会在第一条数据进入的时候自动分析这条数据的情况,给每个value找到最恰当的type,然后以此为该index的mapping。之后再PUT上来的数据,格式如果不符合mapping的,也能存储成功,但是就无法检索了。</p>
<p>mapping中关于store和compress的部分,之前翻译的<a href="http://chenlinux.com/2012/08/26/translate-using-elasticsearch-for-logs">《用ElasticSearch存储日志》</a>已经说的比较详细了。这里我的建议是 disable 掉 <code class="highlighter-rouge">_all</code>,但是 enable 住 <code class="highlighter-rouge">_source</code>!! 经过我的惨痛测试,如果连 <code class="highlighter-rouge">_source</code> 也 disable 掉的话,一旦你重启进程,整个 index 里除了 <code class="highlighter-rouge">_id</code>,<code class="highlighter-rouge">_timestamp</code> 和 <code class="highlighter-rouge">_score</code> 三个默认字段,啥都丢了……</p>
<h1 id="api">API简介</h1>
<p>ES的API,最基本的就是CRUD操作了,这部分是标准的REST,就不说了。</p>
<p>然后还有三个API比较重要且常用,分别是: bulk/count/search。</p>
<ul>
<li>Bulk顾名思义,把多个单条的记录合并成一个大数组统一提交,这样避免一条条发送的header解析,索引频繁更新,indexing速度大大提高</li>
<li>Count根据POST的json,返回命中范围内的总条数。当然没POST时就直接返回该index的总条数了。</li>
<li>Search根据POST的json或者GET的args,返回命中范围内的数据。这是最重要的部分了。下面说说常用的search API:</li>
</ul>
<h2 id="query">query</h2>
<p>一旦使用search,必须至少提供query参数,然后在这个query的基础上进行接下来其他的检索。query参数又分三类:</p>
<ul>
<li><code class="highlighter-rouge">"match_all" : { }</code> 直接请求全部;</li>
<li><code class="highlighter-rouge">"term"/"text"/"prefix"/"wildcard" : { "key" : "value" }</code> 根据字符串搜索(严格相等/片断/前缀/匹配符);</li>
<li><code class="highlighter-rouge">"range" : { "@timestamp" : { "from" : "now-1d", "to" : "now" } }</code> 根据范围搜索,如果type是时间格式,可以使用内置的now表示当前,然后用-1d/h/m/s来往前推。</li>
</ul>
<h2 id="filter">filter</h2>
<p>上面提到的query的参数,在filter中也都存在。此外,还有比较重要的参数就是连接操作:</p>
<ul>
<li><code class="highlighter-rouge">"or"/"and" : [{"range":{}}, {"prefix":""}]</code> 两个filter的查询,交集或者合集;</li>
<li><code class="highlighter-rouge">"bool" : ["must":{},"must_not":{},"should":{}]</code> 上面的and虽然更快,但是只能支持两个,超过两个的,要用 bool 方法;</li>
<li><code class="highlighter-rouge">"not"/"limit" : {}</code> 取反和限定执行数。注意这个limit和mysql什么的有点不同:它限定的是在每个shards上执行多少条。如果你有5个shards,其实对整个index是limit了5倍大小的设定值。</li>
</ul>
<p>另一点比较关键的是:filter结果默认是不缓存的,如果常用,需要指定 <code class="highlighter-rouge">"_cache" : true</code>。</p>
<h2 id="facets">facets</h2>
<p>facets接口可以根据query返回统计数据,最基础的是terms和statistical两种。不过在日志分析的情况下,最常用的是:</p>
<ul>
<li><code class="highlighter-rouge">"histogram" : { "key_field" : "", "value_field" : "", "interval" : "" }</code> 根据时间间隔返回柱状图式的统计数据;</li>
<li><code class="highlighter-rouge">"terms_stats" : { "key_field" : "", "value_field" : "" }</code> 根据key的情况返回value的统计数据,类似group by的意思。</li>
</ul>
<p>这里就涉及到前面mapping里为什么针对每个field都设定type的原因了。因为 <code class="highlighter-rouge">histogram</code> 里的 <code class="highlighter-rouge">key_field</code> 只能是 <code class="highlighter-rouge">dateOptionalTime</code> 格式的,<code class="highlighter-rouge">value_field</code> 只能是 <code class="highlighter-rouge">string</code> 格式的;而 <code class="highlighter-rouge">terms_stats</code> 里的 <code class="highlighter-rouge">key_field</code> 只能是 <code class="highlighter-rouge">string</code> 格式的,<code class="highlighter-rouge">value_field</code> 只能是 <code class="highlighter-rouge">numberic</code> 格式的。</p>
<p>而我们都知道,http code那些200/304/400/503神马的,看起来是数字,我们却需要的是他们的count数据,不是算他们的平均数。所以不能由ES动态的认定为long,得指定为string。</p>
<h1 id="analyze">analyze简介</h1>
<p>对于logstash分析日志,基本没有提到analyze的部分,包括Kibana也是。但是做web日志分析,其实也需要注意analyze。因为ES默认提供并开启了一些analyze。最简单的比如空格分隔表示单词,斜线分割表示url路径,@分割表示email地址等等。文档地址见<a href="http://www.elasticsearch.org/guide/reference/index-modules/analysis/">http://www.elasticsearch.org/guide/reference/index-modules/analysis/</a>。当然ES社区的中国人也有提供中文分词的plugin。通常情况下,analyze工作的很好。嗯,ES比其他全文索引工具在默认情况下都工作的好。</p>
<p>但是当你想算的是今天访问的url排名,或者来访者IP排名的时候,麻烦来了,你苦苦等待N久,最后一看排名是这样的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>jpg 2345678
html 123456
20121021 34567
bbs 9876
</code></pre>
</div>
<p>对,你的url被ES辛辛苦苦的用 / 和 . 分割了,然后每个单词排序来再返回给你。如果你是在一个数千万条的大型库上运行的话,基本吃个饭回来才能有结果。</p>
<p>事实上,url就是一个整体,所以在mapping中,要定义好在indexing的时候,不要启用analyzer。这样,返回一个你心目中想要的正确的url排名,时间从吃个午饭直接缩减到打个喷嚏了!而如果是访问者ip,时间则是眨下眼就够了!</p>
<p>注意:analyze不是只有indexing的时候能用,在query的时候,也可以单独指定某个analyze来分析记录。analyze其实是和search并列的API,不过目前场景下用不上,就不说了。</p>
<p>有以上API,基本上一个针对logstash的ES数据分析系统后台就足够构建出来了。剩下的就是前端页面的事情,这方面可以参考logstash的Kibana,更广义一些的ES数据可视化可以参考ES的blog: <a href="http://www.elasticsearch.cn/blog/2011/05/13/data-visualization-with-elasticsearch-and-protovis.html">http://www.elasticsearch.cn/blog/2011/05/13/data-visualization-with-elasticsearch-and-protovis.html</a>,笔者的译文见<a href="http://chenlinux.com/2012/11/18/data-visualization-with-elasticsearch-and-protovis">http://chenlinux.com/2012/11/18/data-visualization-with-elasticsearch-and-protovis</a>。</p>
<h1 id="section">性能监控</h1>
<p>ES周边的工具有很多。目前我主要用三种方式:</p>
<ul>
<li>es_head: 这个主要提供的是健康状态查询,当然标签页里也提供了简单的form给你提交API请求。es_head现在可以直接通过 <code class="highlighter-rouge">elasticsearch/bin/plugin -install mobz/elasticsearch-head</code> 安装,然后浏览器里直接输入 <code class="highlighter-rouge">http://$eshost:9200/_plugin/head/</code> 就可以看到cluster/node/index/shards的状态了。</li>
<li>bigdesk: 这个主要提供的是节点的实时状态监控,包括jvm的情况,linux的情况,elasticsearch的情况。排查性能问题的时候很有用,现在也可以通过 <code class="highlighter-rouge">elasticsearch/bin/plugin -install lukas-vlcek/bigdesk</code> 直接安装了。然后浏览器里直接输入 <code class="highlighter-rouge">http://$eshost:9200/_plugin/bigdesk/</code> 就可以看到了。注意如果使用的 <code class="highlighter-rouge">bulk_index</code> 的话,如果选择的刷新间隔太长,indexing per second数据是不准的。</li>
<li>然后是最基础的办法,通过ES本身的status API获取状态。因为上面都是web工具,如果想要避免上文提到的故障很久才发现的问题,我们需要一个可以提供给nagios使用的办法,这很简单就可以做到。刚巧ES本身也有green/yellow/red等不同的状态。所以很简单完成一个check_es_health.sh如下:</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#!/bin/sh</span>
<span class="nv">ES_HOST</span><span class="o">=</span><span class="nv">$1</span>
<span class="nv">ES_URI</span><span class="o">=</span><span class="s2">"http://</span><span class="k">${</span><span class="nv">ES_HOST</span><span class="k">}</span><span class="s2">:9200/_cluster/health"</span>
<span class="nv">RES_JSON</span><span class="o">=</span><span class="sb">`</span>curl -s <span class="k">${</span><span class="nv">ES_URI</span><span class="k">}</span><span class="sb">`</span>
<span class="nv">status</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="k">${</span><span class="nv">RES_JSON</span><span class="k">}</span>|awk -F<span class="se">\"</span> <span class="s1">'{print $8}'</span><span class="sb">`</span>
<span class="nv">failed</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="k">${</span><span class="nv">RES_JSON</span><span class="k">}</span>|awk -F<span class="se">\"</span> <span class="s1">'{print $NF}'</span>|sed <span class="s1">'s/^:\([0-9]*\)}/\1/'</span><span class="sb">`</span>
<span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="nv">$status</span><span class="s2">"</span> -eq <span class="s2">"green"</span> <span class="o">]]</span>;<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"ES Cluster OK | failed_node=</span><span class="k">${</span><span class="nv">failed</span><span class="k">}</span><span class="s2">"</span>
<span class="nb">exit </span>0
<span class="k">elif</span> <span class="o">[[</span> <span class="s2">"</span><span class="nv">$status</span><span class="s2">"</span> -eq <span class="s2">"yellow"</span> <span class="o">]]</span>;<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Warning! ES Cluster shards relocating or initializing. | failed_node=</span><span class="k">${</span><span class="nv">failed</span><span class="k">}</span><span class="s2">"</span>
<span class="nb">exit </span>1
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"Critical! ES Cluster shards unassigned. | failed_node=</span><span class="k">${</span><span class="nv">failed</span><span class="k">}</span><span class="s2">"</span>
<span class="nb">exit </span>2
<span class="k">fi</span>
</code></pre>
</div>
<h1 id="section-1">其他插件</h1>
<p>ES是一个很活跃的开源项目,所以如果有其他目前ES没有你有觉得有需要的功能,大可以上github搜索一下,或许别人早已经做完相关插件了。</p>
<p>比如我就在上面找到一个plugin叫elasticfacets。加强了ES的 <code class="highlighter-rouge">date_histogram</code> 功能,原先只能针对某个 <code class="highlighter-rouge">value_field</code> 做攻击,这个plugin可以在这个基础上,把 <code class="highlighter-rouge">value_field</code> 加强成又一层facets。项目地址: <a href="https://github.com/bleskes/elasticfacets">https://github.com/bleskes/elasticfacets</a>。之前和作者反馈了在ES 0.19.8上的问题,不知道修复没。或许最好还是用0.19.9吧。</p>
<h1 id="section-2">邮件列表</h1>
<p>ES的邮件列表基本每天都有四五十封邮件,地址是:<a href="mailto:elasticsearch@googlegroups.com">elasticsearch@googlegroups.com</a>。</p>
用Juggernaut实时推送syslog分析结果
2012-10-17T00:00:00+08:00
web
syslog
javascript
websocket
redis
perl
http://chenlinux.com/2012/10/17/juggernaut-for-syslog-check
<p>大家一般都会用rsyslog或者syslog-ng之类的收集系统日志。不过收集之后的处理就各种各样了。这里提供一个简单的处理,按日期保存成文件,然后定时分析新增内容,通过websocket推送到页面报警。这对于像磁盘错误等信息比较有用。因为等nagios之类的监控反应出来,故障可能就已经到你措手不及的地步了。</p>
<p>这里介绍一下Juggernaut项目,作者当初还在上学的时候就开始搞这个项目并最终因为这个项目找到的工作。不过几个月前他宣布不再维护了,因为他觉得html5已经普及,大家直接写websocket就够了。额,在口年的中国,我觉得Juggernaut还是很有意义的。项目地址:<a href="https://github.com/maccman/juggernaut">https://github.com/maccman/juggernaut</a>。</p>
<p>举例中有ruby、js和python的样例,不过既然是用Redis传消息,那么改成perl的代码跟python的比差距也就很小了。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/perl</span>
<span class="k">use</span> <span class="nv">AnyEvent</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Getopt::</span><span class="nv">Long</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Redis</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">JSON</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">POSIX</span> <span class="sx">qw/ strftime /</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="mf">5.010</span><span class="p">;</span>
<span class="c1"># 后台运行</span>
<span class="k">use</span> <span class="nn">App::</span><span class="nv">Daemon</span> <span class="sx">qw( daemonize )</span><span class="p">;</span>
<span class="nv">$</span><span class="nn">App::Daemon::</span><span class="nv">as_user</span> <span class="o">=</span> <span class="s">"root"</span><span class="p">;</span>
<span class="nv">daemonize</span><span class="p">();</span>
<span class="c1"># 设定调试和间隔</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$debug</span><span class="p">,</span> <span class="nv">$interval</span><span class="p">,</span> <span class="nv">$help</span><span class="p">);</span>
<span class="nv">GetOptions</span><span class="p">(</span>
<span class="s">"debug|d"</span> <span class="o">=></span> <span class="o">\</span><span class="nv">$debug</span><span class="p">,</span>
<span class="s">"interval|i=i"</span> <span class="o">=></span> <span class="o">\</span><span class="nv">$interval</span><span class="p">,</span>
<span class="s">"help|h"</span> <span class="o">=></span> <span class="o">\</span><span class="nv">$help</span><span class="p">,</span>
<span class="p">);</span>
<span class="k">if</span><span class="p">(</span> <span class="nv">$help</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">say</span> <span class="s">"Usage: $0 [start|stop|-X] -d -i num"</span><span class="p">;</span>
<span class="nv">say</span> <span class="s">" -X means run frontend;"</span><span class="p">;</span>
<span class="nv">say</span> <span class="s">" -d means debug for timer and submit;"</span><span class="p">;</span>
<span class="nv">say</span> <span class="s">" -i means define a special interval seconds for regexp and submit, default set 300s."</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">$interval</span> <span class="o">=</span> <span class="mi">300</span> <span class="k">unless</span> <span class="nv">$interval</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@str</span> <span class="o">=</span> <span class="p">();</span>
<span class="c1"># syslog的pri,定义见http://www.ietf.org/rfc/rfc3164.txt。因为本例只收集kernel信息,即Facility = 0,所以PRI = 0 * 8 + Severity。这里为了页面好看直接写成bootstrap里button的class了。</span>
<span class="k">my</span> <span class="nv">@pri</span> <span class="o">=</span> <span class="sx">qw(
btn-inverse
btn-danger
btn-warning
btn-success
btn-info
btn-primary
btn
disabled
)</span><span class="p">;</span>
<span class="c1"># syslog格式<IP> <TIME> <PRI> kernel: <MSG></span>
<span class="c1"># 注意最后msg里的\S.+,因为当内存出错等情况时,msg里开头会以空格表示附属关系</span>
<span class="k">my</span> <span class="nv">$re</span> <span class="o">=</span> <span class="sx">qr/((?:\d{1,3}\.){3}\d{1,3}) \[(\w+ \d+ \d{2}:\d{2}:\d{2})\] <(\d+)> kernel: (\S.+)/</span><span class="p">;</span>
<span class="c1"># 目前是直接收集成时间格式命名了,所以每天crond里要restart脚本,如果是日志名不变,crond切割的,那么脚本可以一直跑</span>
<span class="nb">open</span><span class="p">(</span><span class="k">my</span> <span class="nv">$r</span><span class="p">,</span> <span class="s">"-|"</span><span class="p">,</span> <span class="s">"tail"</span><span class="p">,</span> <span class="s">"-F"</span><span class="p">,</span> <span class="s">'/data1/syslog/kern.'</span> <span class="o">.</span> <span class="nv">strftime</span><span class="p">(</span><span class="s">"%Y%m%d"</span><span class="p">,</span> <span class="nb">localtime</span><span class="p">)</span> <span class="o">.</span> <span class="s">'.log'</span> <span class="p">)</span> <span class="ow">or</span> <span class="nb">die</span> <span class="s">"can't fork: $!"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$io</span> <span class="o">=</span> <span class="nv">AnyEvent</span><span class="o">-></span><span class="nv">io</span><span class="p">(</span>
<span class="nv">fh</span> <span class="o">=></span> <span class="nv">$r</span><span class="p">,</span>
<span class="nv">poll</span> <span class="o">=></span> <span class="s">"r"</span><span class="p">,</span>
<span class="nv">cb</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$input</span> <span class="o">=</span> <span class="nb">scalar</span> <span class="sr"><$r></span><span class="p">;</span>
<span class="k">return</span> <span class="k">if</span> <span class="nv">$input</span> <span class="o">=~</span> <span class="sr">/repeat|suppress|window/</span><span class="p">;</span>
<span class="nb">push</span> <span class="nv">@str</span><span class="p">,</span> <span class="nv">$input</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">);</span>
<span class="k">my</span> <span class="nv">$w</span> <span class="o">=</span> <span class="nv">AnyEvent</span><span class="o">-></span><span class="nv">timer</span><span class="p">(</span>
<span class="nv">after</span> <span class="o">=></span> <span class="nv">$interval</span><span class="p">,</span>
<span class="nv">interval</span> <span class="o">=></span> <span class="nv">$interval</span><span class="p">,</span>
<span class="nv">cb</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$data</span><span class="p">;</span>
<span class="nv">say</span> <span class="s">"######## "</span><span class="p">,</span><span class="nb">time</span> <span class="k">if</span> <span class="nv">$debug</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span> <span class="nv">@str</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">next</span> <span class="k">unless</span> <span class="nv">$_</span> <span class="o">=~</span> <span class="nv">$re</span><span class="p">;</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$ip</span><span class="p">,</span> <span class="nv">$time</span><span class="p">,</span> <span class="nv">$level</span><span class="p">,</span> <span class="nv">$msg</span> <span class="p">)</span> <span class="o">=</span> <span class="p">(</span> <span class="nv">$1</span><span class="p">,</span> <span class="nv">$2</span><span class="p">,</span> <span class="nv">$3</span><span class="p">,</span> <span class="nv">$4</span> <span class="p">);</span>
<span class="k">next</span> <span class="k">if</span> <span class="nb">exists</span> <span class="nv">$data</span><span class="o">-></span><span class="p">{</span><span class="nv">$ip</span><span class="p">};</span>
<span class="nv">$data</span><span class="o">-></span><span class="p">{</span><span class="nv">$ip</span><span class="p">}</span> <span class="o">=</span> <span class="nv">classify</span><span class="p">(</span><span class="nv">$msg</span><span class="p">);</span>
<span class="nv">submit</span><span class="p">(</span> <span class="nv">$time</span><span class="p">,</span> <span class="nv">$ip</span><span class="p">,</span> <span class="nv">$pri</span><span class="p">[</span><span class="nv">$level</span><span class="p">],</span> <span class="nv">$data</span><span class="o">-></span><span class="p">{</span><span class="nv">$ip</span><span class="p">},</span> <span class="nv">$msg</span> <span class="p">);</span>
<span class="p">};</span>
<span class="nv">@str</span> <span class="o">=</span> <span class="p">();</span>
<span class="p">},</span>
<span class="p">);</span>
<span class="nv">AnyEvent</span><span class="o">-></span><span class="nv">condvar</span><span class="o">-></span><span class="nb">recv</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">submit</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">@msg</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="nv">say</span> <span class="nv">@msg</span> <span class="k">if</span> <span class="nv">$debug</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$redis</span> <span class="o">=</span> <span class="nv">Redis</span><span class="o">-></span><span class="k">new</span><span class="p">(</span> <span class="nv">server</span> <span class="o">=></span> <span class="s">'198.168.0.2:6379'</span> <span class="p">);</span>
<span class="nv">$redis</span><span class="o">-></span><span class="nv">publish</span><span class="p">(</span><span class="s">"juggernaut"</span><span class="p">,</span> <span class="nv">to_json</span><span class="p">({</span>
<span class="nv">channels</span> <span class="o">=></span> <span class="p">[</span><span class="s">'channel1'</span><span class="p">],</span>
<span class="nv">data</span> <span class="o">=></span> <span class="o">\</span><span class="nv">@msg</span><span class="p">,</span>
<span class="p">}));</span>
<span class="p">};</span>
<span class="k">sub </span><span class="nf">classify</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$msg</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$msg</span> <span class="o">=~</span> <span class="sr">/TCP|UDP|SYN|socket/</span> <span class="p">)</span> <span class="p">{</span>
<span class="s">'network'</span><span class="p">;</span>
<span class="p">}</span> <span class="k">elsif</span> <span class="p">(</span> <span class="nv">$msg</span> <span class="o">=~</span> <span class="sr">/segfault|swap|mem|allocation/</span> <span class="p">)</span> <span class="p">{</span>
<span class="s">'memory'</span><span class="p">;</span>
<span class="p">}</span> <span class="k">elsif</span> <span class="p">(</span> <span class="nv">$msg</span> <span class="o">=~</span> <span class="sr">/IPMI|EXT|cciss|scsi|mpt|usb|DRAC|sd\w/</span> <span class="p">)</span> <span class="p">{</span>
<span class="s">'disk'</span><span class="p">;</span>
<span class="p">}</span> <span class="k">elsif</span> <span class="p">(</span> <span class="nv">$msg</span> <span class="o">=~</span> <span class="sr">/CPU|IRQ/</span> <span class="p">)</span> <span class="p">{</span>
<span class="s">'cpu'</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="s">'unknown'</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">};</span>
</code></pre>
</div>
<p>然后写html页面来接收。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nt"><html></span>
<span class="nt"><head></span>
<span class="nt"><meta</span> <span class="na">name=</span><span class="s">"charset"</span> <span class="na">content=</span><span class="s">"utf-8"</span><span class="nt">></span>
<span class="nt"><title></span>syslog-push-webUI<span class="nt"></title></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"http://192.168.0.2:8080/application.js"</span> <span class="na">type=</span><span class="s">"text/javascript"</span> <span class="na">charset=</span><span class="s">"utf-8"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"/javascripts/jquery-1.7.2.min.js "</span> <span class="na">type=</span><span class="s">"text/javascript"</span> <span class="na">charset=</span><span class="s">"utf-8"</span><span class="nt">></script></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><div</span> <span class="na">id=</span><span class="s">'syslog'</span> <span class="na">style=</span><span class="s">"overflow-y: scroll; border: #999"</span><span class="nt">></span>
<span class="nt"><ul</span> <span class="na">class=</span><span class="s">"unstyled"</span> <span class="na">id=</span><span class="s">'msg'</span><span class="nt">></span>
<span class="nt"></ul></span>
<span class="nt"></div></span>
<span class="nt"><div><button</span> <span class="na">class=</span><span class="s">"btn"</span> <span class="na">id=</span><span class="s">"notify-permission-button"</span><span class="nt">></span>开启桌面通知<span class="nt"></button></div></span>
<span class="nt"><script </span><span class="na">type=</span><span class="s">"text/javascript"</span> <span class="na">charset=</span><span class="s">"utf-8"</span><span class="nt">></span>
<span class="nx">$</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">log</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">data</span><span class="p">){</span>
<span class="kd">var</span> <span class="nx">msg</span><span class="p">;</span>
<span class="nx">$</span><span class="p">(</span><span class="s1">'#msg li:gt(40)'</span><span class="p">).</span><span class="nx">remove</span><span class="p">();</span>
<span class="k">if</span><span class="p">(</span> <span class="k">typeof</span><span class="p">(</span><span class="nx">data</span><span class="p">)</span> <span class="o">==</span> <span class="s1">'string'</span> <span class="p">)</span> <span class="p">{</span>
<span class="nx">msg</span> <span class="o">=</span> <span class="nx">data</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">msg</span> <span class="o">=</span> <span class="s1">'<blockquote>'</span><span class="p">;</span>
<span class="nx">msg</span> <span class="o">+=</span> <span class="s1">'<button type = "button" class="btn-mini '</span> <span class="o">+</span> <span class="nx">data</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="o">+</span> <span class="s1">'">'</span> <span class="o">+</span> <span class="nx">data</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span> <span class="o">+</span> <span class="s1">'</button>: '</span><span class="p">;</span>
<span class="nx">msg</span> <span class="o">+=</span> <span class="s1">'<code>'</span> <span class="o">+</span> <span class="nx">data</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span> <span class="o">+</span> <span class="s1">'</code>'</span><span class="p">;</span>
<span class="nx">msg</span> <span class="o">+=</span> <span class="s1">'<small><strong>'</span> <span class="o">+</span> <span class="nx">data</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="s1">'</strong> '</span><span class="p">;</span>
<span class="nx">msg</span> <span class="o">+=</span> <span class="s1">'<cite>'</span> <span class="o">+</span> <span class="nx">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">+</span> <span class="s1">'</cite></small>'</span><span class="p">;</span>
<span class="nx">msg</span> <span class="o">+=</span> <span class="s1">'</blockquote>'</span><span class="p">;</span>
<span class="p">}</span>
<span class="nx">$</span><span class="p">(</span><span class="s1">'#msg:first-child'</span><span class="p">).</span><span class="nx">prepend</span><span class="p">(</span><span class="s1">'<li>'</span> <span class="o">+</span> <span class="nx">msg</span> <span class="o">+</span> <span class="s1">'</li>'</span><span class="p">);</span>
<span class="p">};</span>
<span class="kd">var</span> <span class="nx">jug</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Juggernaut</span><span class="p">({</span>
<span class="na">secure</span><span class="p">:</span> <span class="p">(</span><span class="s1">'https:'</span> <span class="o">==</span> <span class="nb">document</span><span class="p">.</span><span class="nx">location</span><span class="p">.</span><span class="nx">protocol</span><span class="p">),</span>
<span class="na">host</span><span class="p">:</span> <span class="nb">document</span><span class="p">.</span><span class="nx">location</span><span class="p">.</span><span class="nx">hostname</span><span class="p">,</span>
<span class="na">port</span><span class="p">:</span> <span class="mi">8080</span> <span class="o">||</span> <span class="nb">document</span><span class="p">.</span><span class="nx">location</span><span class="p">.</span><span class="nx">port</span>
<span class="p">});</span>
<span class="nx">jug</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"connect"</span><span class="p">,</span> <span class="kd">function</span><span class="p">(){</span> <span class="nx">log</span><span class="p">(</span><span class="s2">"<code>Connected</code>"</span><span class="p">)</span> <span class="p">});</span>
<span class="nx">jug</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"disconnect"</span><span class="p">,</span> <span class="kd">function</span><span class="p">(){</span> <span class="nx">log</span><span class="p">(</span><span class="s2">"<code>Disconnected</code>"</span><span class="p">)</span> <span class="p">});</span>
<span class="nx">jug</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"reconnect"</span><span class="p">,</span> <span class="kd">function</span><span class="p">(){</span> <span class="nx">log</span><span class="p">(</span><span class="s2">"<code>Reconnecting</code>"</span><span class="p">)</span> <span class="p">});</span>
<span class="nx">jug</span><span class="p">.</span><span class="nx">subscribe</span><span class="p">(</span><span class="s2">"channel1"</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">data</span><span class="p">){</span>
<span class="nx">log</span><span class="p">(</span><span class="nx">data</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span> <span class="nx">data</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="o">!=</span> <span class="s1">'btn'</span> <span class="p">)</span> <span class="p">{</span>
<span class="nx">desk_notify</span><span class="p">(</span><span class="nx">data</span><span class="p">);</span>
<span class="p">};</span>
<span class="p">});</span>
<span class="kd">var</span> <span class="nx">desk_notify</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">data</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="nb">window</span><span class="p">.</span><span class="nx">webkitNotifications</span><span class="p">){</span>
<span class="k">if</span> <span class="p">(</span><span class="nb">window</span><span class="p">.</span><span class="nx">webkitNotifications</span><span class="p">.</span><span class="nx">checkPermission</span><span class="p">()</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">RequestPermission</span><span class="p">(</span><span class="nx">desk_notify</span><span class="p">(</span><span class="nx">data</span><span class="p">));</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">notification</span> <span class="o">=</span> <span class="nx">webkitNotifications</span><span class="p">.</span><span class="nx">createNotification</span><span class="p">(</span>
<span class="s1">'http://a.xnimg.cn/imgpro/app/mobile/renren_phone_icon2.png'</span><span class="p">,</span>
<span class="nx">data</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span>
<span class="nx">data</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span>
<span class="p">);</span>
<span class="nx">notification</span><span class="p">.</span><span class="nx">show</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">RequestPermission</span><span class="p">(</span><span class="nx">callback</span><span class="p">)</span> <span class="p">{</span>
<span class="nb">window</span><span class="p">.</span><span class="nx">webkitNotifications</span><span class="p">.</span><span class="nx">requestPermission</span><span class="p">(</span><span class="nx">callback</span><span class="p">);</span>
<span class="p">}</span>
<span class="nx">$</span><span class="p">(</span><span class="s1">'#notify-permission-button'</span><span class="p">).</span><span class="nx">click</span><span class="p">(</span><span class="kd">function</span><span class="p">(){</span>
<span class="nx">$</span><span class="p">(</span><span class="s1">'#notify-permission-button'</span><span class="p">).</span><span class="nx">hide</span><span class="p">();</span>
<span class="nx">desk_notify</span><span class="p">([</span><span class="s1">''</span><span class="p">,</span><span class="s1">'syslog realtime push'</span><span class="p">,</span><span class="s1">''</span><span class="p">,</span><span class="s1">''</span><span class="p">,</span><span class="s1">'开启'</span><span class="p">]);</span>
<span class="p">});</span>
<span class="p">});</span>
<span class="nt"></script></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</code></pre>
</div>
<p>这里除了juggernaut的代码以外,还加上了chrome独有的webkitnotification功能,这样使用chrome的话,可以打开桌面通知,监控效果更佳~</p>
<p>注意:chrome桌面通知的权限授予,不能通过页面代码自动触发,必须显式的用button.click来触发。</p>
<p><strong>2013 年 2 月 17 日更新:</strong></p>
<p>我的 Ubuntu 12.10 更新到 firefox 18.0.2 版本后,以上代码也可以出现桌面通知了。不过奇怪的是:第一至今没有看到哪里有这个更新说明;第二我确实没有安装相关的extension,事实上我一共就安装了 firebug/xmarks/adblocks 三个扩展。</p>
Perl5里的gather/take
2012-10-02T00:00:00+08:00
perl
http://chenlinux.com/2012/10/02/gather-take-in-perl5
<p>九月末的YAPC::Asia上,Larry Wall展示了一下怎么把一个perl5上很标准的排序脚本改造成perl6脚本。主要是条件语句不再用()了,子函数传参方式,对象化操作等等。唯独有个命令是之前未见过的:gather/take。用这个可以减少临时变量的使用。</p>
<p>找了一下,类似命令在perl5中有多个模块对应:<a href="http://search.cpan.org/~moritz/Perl6-GatherTake-0.0.3/lib/Perl6/GatherTake.pm">Perl6::GatherTake</a>、<a href="http://search.cpan.org/~gaal/Perl6-Take-0.04/lib/Perl6/Take.pm">Perl6::Take</a>、<a href="http://search.cpan.org/~dconway/Perl6-Gather-0.42/lib/Perl6/Gather.pm">Perl6::Gather</a>、<a href="http://search.cpan.org/~flora/List-Gather-0.06/lib/List/Gather.pm">List::Gather</a>和<a href="http://search.cpan.org/~frew/Syntax-Keyword-Gather-1.002000/lib/Syntax/Keyword/Gather.pm">Syntax::Keyword::Gather</a>。</p>
<p>具体的说明,可以看<a href="http://search.cpan.org/~dconway/Perl6-Gather-0.42/lib/Perl6/Gather.pm">Perl6::Gather</a>的POD说明,比较详细。其他的都是简单给个例子就是了。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nn">Perl6::</span><span class="nv">Gather</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">gather</span> <span class="p">{</span>
<span class="k">for</span> <span class="p">(</span><span class="nv">@data</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">take</span> <span class="k">if</span> <span class="nv">$_</span> <span class="nv">%</span> <span class="nv">2</span><span class="p">;</span>
<span class="nv">take</span> <span class="nv">to_num</span><span class="p">(</span><span class="nv">$_</span><span class="p">)</span> <span class="k">if</span> <span class="sr">/(?:one|three|five|nine)\z/</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">take</span> <span class="mi">1</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">7</span><span class="p">,</span><span class="mi">9</span> <span class="k">unless</span> <span class="nv">gathered</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
<p>相当于:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">my</span> <span class="nv">@arrays</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="nv">@data</span><span class="p">)</span> <span class="p">{</span>
<span class="nb">push</span> <span class="nv">@arrays</span><span class="p">,</span> <span class="nv">$_</span> <span class="k">if</span> <span class="nv">$_</span> <span class="nv">%</span> <span class="nv">2</span><span class="p">;</span>
<span class="nb">push</span> <span class="nv">@arrays</span><span class="p">,</span> <span class="nv">to_num</span><span class="p">(</span><span class="nv">$_</span><span class="p">)</span> <span class="k">if</span> <span class="sr">/(?:one|three|five|nine)\z/</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">@arrays</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">7</span><span class="p">,</span><span class="mi">9</span><span class="p">)</span> <span class="k">unless</span> <span class="nv">@arrays</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">@arrays</span><span class="p">;</span>
</code></pre>
</div>
<p>省略的就是这个push用的临时@arrays变量。同样也可以是标量,用~gather就可以省略.=了,比gather前面多一个波浪号~。</p>
用javascript操作新版本amcharts
2012-09-26T00:00:00+08:00
web
amcharts
javascript
http://chenlinux.com/2012/09/26/jsonp-for-new-version-amcharts
<p>新版本的amcharts用js和html5改写。不再简单的用settings.xml而是写成js的object了。好在例子依然详细。下面贴一段从数据库里取值并绘制成多栏图式的代码:<br />
```javascript</p>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>amStock Example</title>
<link rel="stylesheet" href="../amcharts/style.css" type="text/css" />
<script src="../javascripts/jquery-1.7.2.min.js" type="text/javascript"></script>
<script src="../amcharts/amstock.js" type="text/javascript"></script>
<script type="text/javascript">
var chart;
var dataProvider = [];
AmCharts.ready(function() {
$.ajax({
type: "GET",
dataType:"jsonp",
url: "http://api.domain.com/database/table/find",
data: {"body":'{"sort":[["date",-1]], "limit":10000}'},
jsonp:'jsonpCallback',
jsonpCallback:'jsonCallbackFunction',
success: function(data){
},
error: function(jqXHR, textStatus, errorThrown){
alert(jqXHR);
alert(textStatus);
alert(errorThrown);
},
});
});
function jsonCallbackFunction(data) {
parseJSON(data);
createStockChart();
}
function createStockChart() {
chart = new AmCharts.AmStockChart();
chart.pathToImages = "../amcharts/images/";
var categoryAxesSettings = new AmCharts.CategoryAxesSettings();
//定义显示的最小时间段,前提是X轴是Date对象
categoryAxesSettings.minPeriod = "hh";
chart.categoryAxesSettings = categoryAxesSettings;
var dataSet1 = new AmCharts.DataSet();
//注意这里的field要和json里的key一致
dataSet1.fieldMappings = [{
fromField: "Total",
toField: "Total"
}, {
fromField: "Error",
toField: "Error"
}, {
fromField: "ErrorRate",
toField: "ErrorRate",
}];
dataSet1.dataProvider = dataProvider;
dataSet1.categoryField = "date";
chart.dataSets = [dataSet1];
stockPanel1 = new AmCharts.StockPanel();
//第一栏占全图的百分比
stockPanel1.percentHeight = 70;
var valueAxis1 = new AmCharts.ValueAxis();
//Y轴数值位置,amstock中无法设定Y轴偏移量
valueAxis1.position = "right";
valueAxis1.axisColor = "#999999";
stockPanel1.addValueAxis(valueAxis1);
var graph1 = new AmCharts.StockGraph();
graph1.valueField = "Total";
graph1.title = "总计";
//平滑曲线
graph1.type = "smoothedLine";
graph1.lineColor = "#999999";
//曲线下方区域的透明度
graph1.fillAlphas = 0.2;
graph1.useDataSetColors = false;
stockPanel1.addStockGraph(graph1);
var stockLegend1 = new AmCharts.StockLegend();
stockPanel1.stockLegend = stockLegend1;
stockPanel1.drawingIconsEnabled = true;
var valueAxis2 = new AmCharts.ValueAxis();
valueAxis2.axisColor = "#FCD202";
valueAxis2.gridAlpha = 0;
valueAxis2.axisThickness = 2;
stockPanel1.addValueAxis(valueAxis2);
var graph2 = new AmCharts.StockGraph();
graph2.valueAxis = valueAxis2;
graph2.type = "smoothedLine";
graph2.title = "错误";
graph2.valueField = "Error";
//绘点的形状
graph2.bullet = "square";
//图上超过多少点就不显示形状了
graph2.hideBulletsCount = 30;
graph2.lineColor = "#FCD202";
graph2.lineThickness = 3;
graph2.useDataSetColors = false;
stockPanel1.addStockGraph(graph2);
stockPanel2 = new AmCharts.StockPanel();
stockPanel2.percentHeight = 30;
stockPanel2.marginTop = 1;
stockPanel2.categoryAxis.dashLength = 5;
stockPanel2.showCategoryAxis = false;
valueAxis5 = new AmCharts.ValueAxis();
valueAxis5.dashLength = 5;
valueAxis5.gridAlpha = 0;
valueAxis5.axisThickness = 2;
stockPanel2.addValueAxis(valueAxis5);
var graph5 = new AmCharts.StockGraph();
graph5.valueAxis = valueAxis5;
graph5.valueField = "ErrorRate";
graph5.title = "错误比率";
//漂浮显示的文字,可用graph.valueField的[[value]]和graph.descriptionField的[[description]]
graph5.balloonText = "[[value]]%";
graph5.type = "column";
graph5.cornerRadiusTop = 4;
graph5.fillAlphas = 1;
graph5.lineColor = "#FCD202";
graph5.useDataSetColors = false;
stockPanel2.addStockGraph(graph5);
var stockLegend2 = new AmCharts.StockLegend();
stockPanel2.stockLegend = stockLegend2;
chart.panels = [stockPanel1, stockPanel2];
var sbsettings = new AmCharts.ChartScrollbarSettings();
//根据graph1显示拖动条栏
sbsettings.graph = graph1;
sbsettings.graphType = "line";
sbsettings.height = 30;
chart.chartScrollbarSettings = sbsettings;
var cursorSettings = new AmCharts.ChartCursorSettings();
//光标移动时跟随显示漂浮气球
cursorSettings.valueBalloonsEnabled = true;
chart.chartCursorSettings = cursorSettings;
var periodSelector = new AmCharts.PeriodSelector();
periodSelector.position = "bottom";
periodSelector.periods = [{
period: "DD",
selected: true,
count: 1,
label: "1 day"
}, {
period: "DD",
count: 7,
label: "1 week"
}, {
period: "MM",
count: 1,
label: "1 month"
}, {
period: "YYYY",
count: 1,
label: "1 year"
}, {
period: "YTD",
label: "YTD"
}, {
period: "MAX",
label: "MAX"
}];
chart.periodSelector = periodSelector;
chart.write("chartdiv");
};
function parseDate(dateString) {
var dateArray = dateString.split("-");
var date = new Date(Number(dateArray[0]), Number(dateArray[1]) - 1, Number(dateArray[2]), Number(dateArray[3]));
return date;
}
function parseJSON(data){
dataProvider = data.reverse();
//其余列原样保存,时间列必须把字符串转换成Date对象
for( var i = 0; i < dataProvider.length; i++) {
dataProvider[i].date = parseDate(dataProvider[i].date);
}
};
</script>
</head>
<body>
<div id="chartdiv" style="width: 100%; height: 700px;"></div>
</body>
</html>
<p>```</p>
【Logstash系列】数据格式之json-event
2012-09-21T00:00:00+08:00
logstash
nginx
http://chenlinux.com/2012/09/21/json-event-for-logstash
<p>之前的各种示例中,都没有提到logstash的输入输出格式。看起来就好像logstash比Message::Passing少了decoder/encoder一样。其实logstash也有类似的设定的,这就是format。有三种选择:plain/json/json_event。默认情况下是plain。也就是我们之前的通用做法,传文本给logstash,由logstash转换成json。</p>
<p>logstash社区根据某些应用场景,有相关的cookbook。关于访问日志,有<a href="http://cookbook.logstash.net/recipes/apache-json-logs/">http://cookbook.logstash.net/recipes/apache-json-logs/</a>。这是一个不错的思路!我们可以照葫芦画瓢给nginx也定义一下:<br />
<code class="highlighter-rouge">nginx
logformat json '{"@timestamp":"$time_iso8601",'
'"@source":"$server_addr",'
'"@fields":{'
'"client":"$remote_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":$upstream_response_time,'
'"oh":"$upstream_addr",'
'"domain":"$host",'
'"url":"$uri",'
'"status":"$status"}}';
access_log /data/nginx/logs/access.json json;
</code><br />
这里需要注意的地方是:因为最后需要插入ES的某些field是有double/float类型。所以麻烦来了:一些端口监控工具的请求,状态码为400的,因为直接断开,所以并没有链接上upstream的服务器,其$upstream_response_time变量不存在,记录在日志里是-,这对于数值型是非法的定义。直接把带有400的日志通过file格式输入给logstash的时候,因为这个非法定义会报错,并把这行日志给丢弃掉。那么我们就无法统计400请求的数据了。</p>
<p>这里需要变通一下,我们知道其实所谓的Input::File就等效于tail -F ${path}${filename}(当然其实不是,模块的实际做法是在~/.sincedb里记录上次读取的位置,然后每${stat_interval}秒检查一次内容更新,每${discover_interval}秒检查一次文件描述符变更。也就是说默认其实是每秒读一次,一次几百上千行,这样效率更高)。所以我们可以自己运行tail命令,然后sed修正upstream_response_time后通过管道传递给logstash的Input::STDIN,效果是一样一样的。<br />
新的logstash/agent.conf如下:<br />
<code class="highlighter-rouge">ruby
input {
stdin {
type => "nginx"
format => "json_event"
}
}
output {
amqp {
type => "nginx"
host => "10.10.10.10"
key => "cdn"
name => "logstash"
exchange_type => "direct"
}
}
</code><br />
运行命令如下:<br />
<code class="highlighter-rouge">bash
#!/bin/sh
tail -F /data/nginx/logs/access.json \
| sed 's/upstreamtime":-/upstreamtime":0/' \
| /usr/local/logstash/bin/logstash -f /usr/local/logstash/etc/agent.conf &
</code><br />
这样可以直接省略掉昂贵的Grok操作,同时节约原本的_all/_message/_source_host等等格式的空间。</p>
【Message::Passing系列】Regexp::Log模板匹配变量
2012-09-16T00:00:00+08:00
logstash
perl
message-passing
http://chenlinux.com/2012/09/16/regexp-log-demo-for-nginx
<p>上面调用的Regexp::Log::Nginx是base Regexp::Log的实例,CPAN上已经提供了好些server的log regex,见<a href="http://search.cpan.org/search?query=Regexp%3A%3Alog&mode=all">http://search.cpan.org/search?query=Regexp%3A%3Alog&mode=all</a>。<br />
<code class="highlighter-rouge">perl
#!/usr/bin/perl
package Regexp::Log::Nginx;
use warnings;
use strict;
use base qw( Regexp::Log );
use vars qw( $VERSION %DEFAULT %FORMAT %REGEXP );
%DEFAULT = (
format => '%date %status %remotehost %domain %request %originhost %responsetime %upstreamtime %bytes %referer %useragent %xforwarderfor',
capture => [ 'ts', 'status', 'remotehost', 'url', 'oh', 'responsetime', 'upstreamtime', 'bytes' ],
);
%FORMAT = (
':default' => '%date %status %remotehost %domain %request %originhost %responsetime %upstreamtime %bytes %referer %useragent %xforwarderfor',
);
%REGEXP = (
'%date' => '(?#=date)\[(?#=ts)\d{2}\/\w{3}\/\d{4}(?::\d{2}){3}(?#!ts) [-+]\d{4}\](?#!date)',
'%status' => '(?#=status)\d+(?#!status)',
'%remotehost' => '(?#=remotehost)\S+(?#!remotehost)',
'%domain' => '(?#=domain).*?(?#!domain)',
'%request' => '(?#=request)-|(?#=method)\w+(?#!method) (?#=url).*?(?#!url) (?#=version)HTTP/\d\.\d(?#!version)(?#!request)',
'%originhost' => '(?#=originhost)-|(?#=oh).*?(?#!oh):\d+(?#!originhost)',
'%responsetime' => '(?#=responsetime)-|.*?(?#!responsetime)',
'%upstreamtime' => '(?#=upstreamtime).*?(?#!upstreamtime)',
'%bytes' => '(?#=bytes)\d+(?#!bytes)',
'%referer' => '(?#=referer)\"(?#=ref).*?(?#!ref)\"(?#!referer)',
'%useragent' => '(?#=useragent)\"(?#=ua).*?(?#!ua)\"(?#!useragent)',
'%xforwarderfor' => '(?#=xforwarderfor)\"(?#=xff).*?(?#!xff)\"(?#!xforwarderfor)',
);
1;
</code></p>
【Message::Passing系列】过滤器实例
2012-09-16T00:00:00+08:00
logstash
perl
message-passing
http://chenlinux.com/2012/09/16/message-passing-filter-demo
<p><a href="http://github.com/suretec">Message::Passing</a>是Suretec公司为自己的VoIP业务开发的logstash山寨版。这几个月更新还是比较快的。比之前我刚关注它时改变很大。比如Message::Passing::Output::ElasticSearch已经出来了,还有专门的Message::Passing::Filter::ToLogstash,连命令行方式Message::Passing::Role::Script都采用了MooX::Options构建,相当的OO了。</p>
<p>不过可能是运用环境的区别吧,作者一直在filter方面没有什么动静,可怜巴巴的null/all/key/tologstash这么几个,比起input和output的列表差太远了。而且像logstash里的jls-grok这么最有力的工具没有山寨。</p>
<p>今天有空,稍微写了个例子,可以比较方便的定义类似grok_pattern的方式完成对accesslog的json序列化。不过配置方式比Grok还是麻烦不少,以后真用的话,再考虑config的办法吧,这里主要是为了展示Message::Passing::Filter::XXX的编写:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="c1">#!/usr/bin/perl</span>
<span class="nb">package</span> <span class="nn">Message::Passing::Filter::</span><span class="nv">GrokLike</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Moo</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">MooX::Types::MooseLike::</span><span class="nv">Base</span> <span class="sx">qw/ ArrayRef Str /</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">List::</span><span class="nv">MoreUtils</span> <span class="sx">qw/ uniq /</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">DateTime</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Regexp::Log::</span><span class="nv">Nginx</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">namespace::</span><span class="nv">clean</span> <span class="o">-</span><span class="nv">except</span> <span class="o">=></span> <span class="s">'meta'</span><span class="p">;</span>
<span class="nv">with</span> <span class="s">'Message::Passing::Role::Filter'</span><span class="p">;</span>
<span class="nv">has</span> <span class="nb">format</span> <span class="o">=></span> <span class="p">(</span>
<span class="nv">is</span> <span class="o">=></span> <span class="s">'ro'</span><span class="p">,</span>
<span class="nv">isa</span> <span class="o">=></span> <span class="nv">Str</span><span class="p">,</span>
<span class="nv">default</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="s">'%date %status %remotehost %domain %request %originhost %responsetime %upstreamtime %bytes %referer %useragent %xforwarderfor'</span> <span class="p">},</span>
<span class="p">);</span>
<span class="nv">has</span> <span class="nv">capture</span> <span class="o">=></span> <span class="p">(</span>
<span class="nv">is</span> <span class="o">=></span> <span class="s">'ro'</span><span class="p">,</span>
<span class="nv">isa</span> <span class="o">=></span> <span class="nv">ArrayRef</span><span class="p">,</span>
<span class="nv">default</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="p">[</span> <span class="s">'ts'</span><span class="p">,</span> <span class="s">'status'</span><span class="p">,</span> <span class="s">'remotehost'</span><span class="p">,</span> <span class="s">'url'</span><span class="p">,</span> <span class="s">'oh'</span><span class="p">,</span> <span class="s">'responsetime'</span><span class="p">,</span> <span class="s">'upstreamtime'</span><span class="p">,</span> <span class="s">'bytes'</span> <span class="p">]</span> <span class="p">},</span>
<span class="p">);</span>
<span class="nv">has</span> <span class="nv">_grok</span> <span class="o">=></span> <span class="p">(</span>
<span class="nv">is</span> <span class="o">=></span> <span class="s">'ro'</span><span class="p">,</span>
<span class="nv">lazy</span> <span class="o">=></span> <span class="mi">1</span><span class="p">,</span>
<span class="nv">builder</span> <span class="o">=></span> <span class="s">'_build_grok'</span><span class="p">,</span>
<span class="p">);</span>
<span class="k">sub </span><span class="nf">_build_grok</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$self</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$rln</span> <span class="o">=</span> <span class="nn">Regexp::Log::</span><span class="nv">Nginx</span><span class="o">-></span><span class="k">new</span><span class="p">(</span>
<span class="nb">format</span> <span class="o">=></span> <span class="nv">$self</span><span class="o">-></span><span class="nb">format</span><span class="p">,</span>
<span class="nv">capture</span> <span class="o">=></span> <span class="nv">$self</span><span class="o">-></span><span class="nv">capture</span><span class="p">,</span>
<span class="p">);</span>
<span class="k">return</span> <span class="nv">$rln</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">sub </span><span class="nf">filter</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$self</span><span class="p">,</span> <span class="nv">$message</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@fields</span> <span class="o">=</span> <span class="nv">$self</span><span class="o">-></span><span class="nv">_grok</span><span class="o">-></span><span class="nv">capture</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$re</span> <span class="o">=</span> <span class="nv">$self</span><span class="o">-></span><span class="nv">_grok</span><span class="o">-></span><span class="nv">regexp</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">%data</span><span class="p">;</span>
<span class="nv">@data</span><span class="p">{</span><span class="nv">@fields</span><span class="p">}</span> <span class="o">=</span> <span class="nv">$message</span><span class="o">-></span><span class="p">{</span><span class="s">'@message'</span><span class="p">}</span> <span class="o">=~</span> <span class="sr">m/$re/</span><span class="p">;</span>
<span class="nv">$message</span><span class="o">-></span><span class="p">{</span><span class="s">'@fields'</span><span class="p">}</span> <span class="o">=</span> <span class="p">{</span>
<span class="nv">%data</span><span class="p">,</span>
<span class="nv">responsetime</span> <span class="o">=></span> <span class="nv">$data</span><span class="p">{</span><span class="s">'responsetime'</span><span class="p">}</span> <span class="o">+</span> <span class="mi">0</span><span class="p">,</span>
<span class="nv">upstreamtime</span> <span class="o">=></span> <span class="nv">$data</span><span class="p">{</span><span class="s">'upstreamtime'</span><span class="p">}</span> <span class="o">+</span> <span class="mi">0</span><span class="p">,</span>
<span class="nv">bytes</span> <span class="o">=></span> <span class="nv">$data</span><span class="p">{</span><span class="s">'bytes'</span><span class="p">}</span> <span class="o">+</span> <span class="mi">0</span><span class="p">,</span>
<span class="p">};</span>
<span class="nv">$message</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">true</span><span class="p">;</span>
</code></pre>
</div>
<hr />
<p><strong>2012 年 12 月 30 日附注:</strong></p>
<p>前两天已经把这个模块正规化后上传到 CPAN 上了。把捕获定义成 <code class="highlighter-rouge">.ini</code> 文件,同时还加入了类似 logstash 里的 mutate filter 的功能。模块叫 <code class="highlighter-rouge">Message::Passing::Filter::Regexp</code>。同时上传了一个 <code class="highlighter-rouge">Message::Passing::Output::PocketIO</code> 模块,可以打开一个网页时时接收output。</p>
【Message::Passing系列】客户端收集脚本
2012-09-16T00:00:00+08:00
logstash
perl
message-passing
http://chenlinux.com/2012/09/16/message-passing-agent
<p>最后编写一段日志收集的agent:<br />
<code class="highlighter-rouge">perl
#!/usr/bin/perl
package NginxLogCollector;
use Moo;
use MooX::Options;
use Message::Passing::DSL;
use MooX::Types::MooseLike::Base qw/ Str /;
use namespace::clean -except => [qw( meta _options_data _options_config )];
with 'Message::Passing::Role::Script';
option filename => (
is => 'ro',
isa => Str,
default => sub { '/data/nginx/logs/access.log' },
);
option rabbitmq => (
is => 'ro',
isa => Str,
default => sub { '10.3.18.199' },
);
sub build_chain {
my $self = shift;
message_chain {
output rabbitmq => (
class => 'AMQP',
exchange_name => 'logcollect',
# 目前测试结果,发现Input::AMQP无法接收到非topic的exchange
# CPAN上有关RabbitMQ的模块都是这个哥们写的,POD简略到没有一样,表示无语下
# exchange_type => 'direct',
hostname => $self->rabbitmq,
username => 'guest',
password => 'guest',
);
output debug => (
class => 'STDOUT',
);
encoder("encoder",
class => 'JSON',
output_to => 'rabbitmq',
output_to => 'debug',
);
filter grok => (
class => 'GrokLike',
output_to => 'encoder',
);
filter logstash => (
class => 'ToLogstash',
output_to => 'grok',
);
decoder("decoder",
class => 'JSON',
output_to => 'logstash',
);
input nginxlog => (
class => 'FileTail',
output_to => 'decoder',
filename => $self->filename,
);
};
}
__PACKAGE__->start unless caller;
1;
</code><br />
目前就做到这步,之后从rabbitmq里往elasticsearch写的还没搞。从上面的chain可以很清除的看到和logstash一样的管道思想,input->decoder->filter->encoder->output。巧的是两种写法中,decode/encode那个写法跟puppet的DSL定义特别的像。哈哈~</p>
<p>继续贴汇总入库的agent代码:<br />
```perl<br />
#!/usr/bin/perl<br />
use Moo;<br />
use MooX::Options;<br />
use Message::Passing::DSL;<br />
use MooX::Types::MooseLike::Base qw/ Str /;<br />
use namespace::clean -except => [qw( meta _options_data _options_config )];</p>
<p>with ‘Message::Passing::Role::Script’;</p>
<p>option elasticsearch_servers => (<br />
is => ‘ro’,<br />
isa => Str,<br />
default => sub { ‘10.3.18.199:9200’ },<br />
);</p>
<p>option rabbitmq => (<br />
is => ‘ro’,<br />
isa => Str,<br />
default => sub { ‘10.3.18.199’ },<br />
);</p>
<p>sub build_chain {<br />
my $self = shift;<br />
message_chain {<br />
output elasticsearch => (<br />
class => ‘ElasticSearch’,<br />
elasticsearch_servers => [$self->elasticsearch_servers],<br />
);<br />
decoder decoder => (<br />
class => ‘JSON’,<br />
output_to => ‘elasticsearch’,<br />
);<br />
input rabbitmq => (<br />
class => ‘AMQP’,<br />
exchange_name => ‘logcollect’,<br />
queue_name => ‘logcollect’,<br />
hostname => $self->rabbitmq,<br />
username => ‘guest’,<br />
password => ‘guest’,<br />
output_to => ‘decoder’,<br />
);<br />
};<br />
}<br />
<strong>PACKAGE</strong>->start unless caller;<br />
1;<br />
```<br />
总的来说,原有模块相互之间都有些不太协调,用的时候还是要自己改改。比如这里,原版的Output::ElasticSearch是吧之前传递过来的整个message放进@fields里的,这跟Filter::ToLogstash的message结构是完全冲突的。所以需要修改Output::ElasticSearch里的bulk_index()的data=>$data,就可以了,不要多改动。</p>
<p>整个代码变动,参加个人github:<a href="https://github.com/chenryn/Message-Passing.git">https://github.com/chenryn/Message-Passing.git</a></p>
【Message::Passing系列】ElasticSearch的bulk_index速度测试
2012-09-16T00:00:00+08:00
logstash
elasticsearch
perl
message-passing
http://chenlinux.com/2012/09/16/elasticsearch-bulk-index-speed-testing
<p>连续尝试了logstash的elasticsearch/elasticsearch_http/elasticsearch_river三个putput模块,发现其index/bulk/river三种插入方式的实际运行效果速度居然没有差异。而使用perl脚本测试,单例下index不到300msg/sec,bulk接近2500msg/sec,几乎翻了10倍。</p>
<p>测试脚本如下:<br />
<code class="highlighter-rouge">perl
#!/usr/bin/perl -w
use ElasticSearch;
use JSON;
use Time::HiRes qw/time/;
use Data::Dumper;
#curl -XGET 'http://localhost:9200/logstash-2012.09.14/nginx/_mapping'
my $line = '{
"@timestamp" : "2012-09-04T13:38:59.496888Z",
"@tags" : [],
"@fields" : {
"reqtime" : [
0.016
],
"req" : [
"/fmn056/20120812/1645/tiny_r3N9_236f000036ad118d.jpg"
],
"version" : [
"1.1"
],
"useragent" : [
"\"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)\""
],
"port" : [
"80"
],
"size" : [
2360
],
"client" : [
"210.56.223.176"
],
"upstream" : [
"10.9.18.50"
],
"method" : [
"GET"
],
"referer" : [
"photo.renren.com",
"/photo/420723228/photo-6408408309?psource=3&fromVIP=false"
],
"ZONE" : [
"+0800"
],
"code" : [
200
],
"upstime" : [
0.016
]
},
"@source_path" : "//data/nginx/logs/access.log",
"@source" : "file://DBLYD5-32.opi.com//data/nginx/logs/access.log",
"@message" : "[04/Sep/2012:21:38:59 +0800] 200 210.56.223.176 fmn.rrimg.com GET /fmn056/20120812/1645/tiny_r3N9_236f000036ad118d.jpg HTTP/1.1 10.9.18.50:80 0.016 0.016 2360 \"http://photo.renren.com/photo/420723228/photo-6408408309?psource=3&fromVIP=false\" \"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)\" \"-\"",
"@source_host" : "DBLYD5-32.opi.com",
"@type" : "nginx"
}';
my $hash = from_json($line);
my $elsearch = ElasticSearch->new(
servers => '10.4.16.68:9200',
transport => 'httplite',
max_requests => 10000,
);
my $begin = time;
for ( 1 .. 1000 ) {
my @data;
push @data, { index => { data => $hash } } for 1 .. 20;
$elsearch->bulk(
index => 'logstash-test',
type => 'nginx',
actions => \@data
);
}
print 1000 * 20 / (time - $begin);
</code><br />
注意到这里bulk的数组是20个元素。实验证明超过20个会报出HTTP::Lite的错误(附带提示:ElasticSearch::Transport::*的HTTPLite啊AEHTTP啊的模块都是要另外安装的)。而且在使用Logstash::Outputs::ElasticSearchHTTP时,flush_size的default值100也是无法使用的,也是改到20后才行。</p>
<p>ps: 不知道为什么一起发github显示不了,只好拆开了,第一篇,关于ES的index速度测试。</p>
(R)?ex介绍
2012-09-06T00:00:00+08:00
devops
perl
rex
http://chenlinux.com/2012/09/06/intro-rex
<p>按说这文章好像轮不到我写。几个线上运用着的哥们都不出手,我勉强记录一些<a href="http://rexify.org/">官网</a>上没写example但实际应该蛮常用的功能吧:</p>
<ul>
<li>IP列表</li>
</ul>
<p>Rex的示例中,都突出了自己可以方便的对group做集合运算的特点。嗯,这个在早先使用SSH::Batch的时候就很钦佩。但是面对可能某个应用每个节点就一两台设备又有很多应用的情况,在Rexfile里写就很麻烦了。而且难免有其他操作的时候需要用单独的列表。</p>
<p>其实Rex有Rex::Group::Lookup::File模块专门解决这个问题:<br />
<code class="highlighter-rouge">perl
use Rex::Group::Lookup::File;
my $group_path = "/etc/puppet/iplist";
grep {
my $group_file = $_;
my $group_name = $1 if $group_file =~ m#$group_path/(.+)\.list#;
group "$group_name" => lookup_file("$group_file");
} glob("$group_path/*");
</code><br />
比如上面的例子,原先是在puppet上做集群管理的,已经有了现成的一个目录iplist专门存放各个应用的设备ip列表文件。那么在Rexfile开头加上这么两三行代码,就自动把整个应用设备归类到group里了。然后运行rex -Tv就看到Server Groups栏下一串串列表了。</p>
<ul>
<li>错误输出</li>
</ul>
<p>Rex的command示例中,都是如下形式:<br />
<code class="highlighter-rouge">perl
say run 'uptime';
</code><br />
但是大家会发现一个很常见的事情,就是命令行上敲错某个字母了,rex是没有错误提示输出的——当然rex本来的主要目的是做任务管理,理论上你得先保证task写正确。<br />
当然,其实rex也可以返回错误提示的。Rex::Interface::Exec::SSH中对返回结果是有判断的。<br />
<code class="highlighter-rouge">perl
use Rex::Helper::SSH2;
...;
my ($out, $err) = net_ssh2_exec($ssh, "LC_ALL=C $path " . $cmd);
if(wantarray) { return ($out, $err); }
return $out;
</code><br />
对应到run命令的Rex::Commands::Run::run()里则是:<br />
<code class="highlighter-rouge">perl
sub run {
my ($cmd, $code) = @_;
my $path = join(":", Rex::Config->get_path());
my $exec = Rex::Interface::Exec->create;
my ($out, $err) = $exec->exec($cmd, $path);
chomp $out if $out;
chomp $err if $err;
if($code) {
return &$code($out, $err);
}
if(wantarray) {
return split(/\n/, $out);
}
return $out;
}
</code><br />
ok,看到了吧。run命令除了接收cmd外,还可以接收code的。而且也只有code方式可以处理错误输出。这里和exec不同,exec在列表上下文中返回标准输出和错误输出,但run在列表上下文中是把标准输出以行分割成列表元素。</p>
<p>所以要在run下看错误输出的话,应该这么写:<br />
<code class="highlighter-rouge">perl
task "test", group => 'nginx', sub {
run "pa aux|grep ngin[x]", sub {
my ($out, $err) = @_;
print $err ? $err : $out;
};
};
</code></p>
<ul>
<li>Kerberos支持</li>
</ul>
<p>这是个人问题,估计碰到的不会多,姑且记录在此。查阅POD,包括在perl mongers和maillist上询问过了。Net::SSH2用的是libssh2库,这个库确实没办法支持gssapi-keyex/gssapi-micpassword的auth。半年前我在github上问Rex作者,表示有计划提供除Net::SSH2之外的支持,不过时间未定。结果看到他的做法是上个月推出了一个和puppet极类似的http方式的Rex::Endpoint::HTTP,我晕。</p>
<p>浏览了一遍,其实替换Net::SSH2模块并不费劲。于是我花几个小时给rex加上了krb5认证参数,在rex -k或者Rexfile里set -krb5的时候,改用Net::OpenSSH模块来完成。主要一个是connect,因为Net::SSH2是connect和auth分开的;一个是exec,因为Net::SSH2里另建channel的,Net::OpenSSH里直接capture即可;一个是disconnect,同理Net::OpenSSH是没这步直接退出的。至于SFTP,两者都实现了标准的sftp接口,代码甚至一行都不用改就能用(Makefile.PL里还是要加Net::SFTP::Foreign的)。代码见<a href="http://github.com/chenryn/rex">我的fork</a>。</p>
<ul>
<li>集群高性能管理</li>
</ul>
<p>rex中有批量操作的参数,看代码应该是用ForkManager。但中心单机就是单机,所以rex也是在上个月推出了Rex::Gearman模块,通过gearmand把worker作成分布式的工作模式,这样简单有效的完成高性能扩展。gearmand也是我最爱的万能组件了~~</p>
<ul>
<li>命令行集群管理</li>
</ul>
<p>rex的命令行有个怪逻辑,在-e的时候只认-H,-G只读rexfile里的task。但估计很多人都希望得到的是一次定义iplist,然后之后执行任意命令,即-G ‘groupname’ -e “say run ‘commands’“的方式。稍微挪动一下代码段的位置,把elsif(-f $::rexfile){…}改成if(){}并移动到if($opts{‘e’}){…}前面去就好了。本更改已提交个人fork。</p>
【翻译】用ElasticSearch存储日志
2012-08-26T00:00:00+08:00
logstash
elasticsearch
http://chenlinux.com/2012/08/26/translate-using-elasticsearch-for-logs
<h1 id="section">介绍</h1>
<p>如果你使用elasticsearch来存储你的日志,本文给你提供一些做法和建议。</p>
<p>如果你想从多台主机向elasticsearch汇集日志,你有以下多种选择:</p>
<ul>
<li><a href="http://graylog2.org/">Graylog2</a> 安装在一台中心机上,然后它负责往elasticsearch插入日志,而且你可以使用它那个漂亮的搜索界面~</li>
<li><a href="http://logstash.net/">Logstash</a> 他有很多特性,包括你能输入什么日志,如何变换过滤,最好输出到哪里。其中就有输出到elasticsearch,包括直接输出和通过<a href="http://www.elasticsearch.org/guide/reference/river/rabbitmq.html">RabbitMQ的river</a>方式两种。</li>
<li><a href="https://cwiki.apache.org/FLUME/">Apache Flume</a> 这个也可以从海量数据源中获取日志,用”decorators”修改日志,也有各种各样的”sinks”来存储你的输出。和我们相关的是<a href="https://github.com/Aconex/elasticflume">elasticflume sink</a>。</li>
<li>omelasticsearch Rsyslog的输出模块。你可以在你的应用服务器上通过rsyslog直接输出到elasticsearch,也可以用rsyslog传输到中心服务器上来插入日志。或者,两者结合都行。具体如何设置参见<a href="http://wiki.rsyslog.com/index.php/HOWTO:_rsyslog_%2B_elasticsearch">rsyslog Wiki</a>。</li>
<li>定制方案。比如,专门写一个脚本从天南海北的某个服务器传输你的日志到elasticsearch。</li>
</ul>
<p>根据你设定的不同,最佳配置也变化不定。不过总有那么几个有用的指南可以推荐一下:</p>
<h2 id="section-1">内存和打开的文件数</h2>
<p>如果你的elasticsearch运行在专用服务器上,经验值是分配一半内存给elasticsearch。另一半用于系统缓存,这东西也很重要的。</p>
<p>你可以通过修改ES_HEAP_SIZE环境变量来改变这个设定。在启动elasticsearch之前把这个变量改到你的预期值。另一个选择上球该elasticsearch的ES_JAVA_OPTS变量,这个变量时在启动脚本(elasticsearch.in.sh或elasticsearch.bat)里传递的。你必须找到-Xms和-Xmx参数,他们是分配给进程的最小和最大内存。建议设置成相同大小。嗯,ES_HEAP_SIZE其实就是干的这个作用。</p>
<p>你必须确认文件描述符限制对你的elasticsearch足够大,建议值是32000到64000之间。关于这个限制的设置,另有<a href="http://www.elasticsearch.org/tutorials/2011/04/06/too-many-open-files.html">教程</a>可以参见。</p>
<h2 id="section-2">目录数</h2>
<p>一个可选的做法是把所有日志存在一个索引里,然后用<a href="http://www.elasticsearch.org/guide/reference/mapping/ttl-field.html">ttl field</a>来确保就日志被删除掉了。不过当你日志量够大的时候,这可能就是一个问题了,因为用TTL会增加开销,优化这个巨大且唯一的索引需要太长的时间,而且这些操作都是资源密集型的。</p>
<p>建议的办法是基于时间做目录。比如,目录名可以是YYYY-MM-DD的时间格式。时间间隔完全取决于你打算保留多久日志。如果你要保留一周,那一天一个目录就很不错。如果你要保留一年,那一个月一个目录可能更好点。目录不要太多,因为全文搜索的时候开销相应的也会变大。</p>
<p>如果你选择了根据时间存储你的目录,你也可以缩小你的搜索范围到相关的目录上。比如,如果你的大多数搜索都是关于最近的日志的,那么你可以在自己的界面上提供一个”快速搜索”的选项只检索最近的目录。</p>
<h2 id="section-3">轮转和优化</h2>
<p>移除旧日志在有基于时间的目录后变得异常简单:<br />
<code class="highlighter-rouge">bash
$ curl -XDELETE 'http://localhost:9200/old-index-name/'
</code><br />
这个操作的速度非常快,和删除大小差不多的少量文件速度接近。你可以放进crontab里半夜来做。</p>
<p><a href="http://www.elasticsearch.org/guide/reference/api/admin-indices-optimize.html">Optimizing indices</a>是在非高峰时间可以做的一件很不错的事情。因为它可以提高你的搜索速度。尤其是在你是基于时间做目录的情况下,更建议去做了。因为除了当前的目录外,其他都不会再改,你只需要对这些旧目录优化一次就一劳永逸了。<br />
<code class="highlighter-rouge">bash
$ curl -XPOST 'http://localhost:9200/old-index-name/_optimize'
</code></p>
<h2 id="section-4">分片和复制</h2>
<p>通过elasticsearch.yml或者使用REST API,你可以给每个目录配置自己的设定。具体细节参见<a href="http://www.elasticsearch.org/guide/reference/setup/configuration.html">链接</a>。</p>
<p>有趣的是分片和复制的数量。默认情况下,每个目录都被分割成5个分片。如果集群中有一个以上节点存在,每个分片会有一个复制。也就是说每个目录有一共10个分片。当往集群里添加新节点的时候,分片会自动均衡。所以如果你有一个默认目录和11台服务器在集群里的时候,其中一台会不存储任何数据。</p>
<p>每个分片都是一个Lucene索引,所以分片越小,elasticsearch能放进分片新数据越少。如果你把目录分割成更多的分片,插入速度更快。请注意如果你用的是基于时间的目录,你只在当前目录里插入日志,其他旧目录是不会被改变的。</p>
<p>太多的分片带来一定的困难——在空间使用率和搜索时间方面。所以你要找到一个平衡点,你的插入量、搜索频率和使用的硬件条件。</p>
<p>另一方面,复制帮助你的集群在部分节点宕机的时候依然可以运行。复制越多,必须在线运行的节点数就可以越小。复制在搜索的时候也有用——更多的复制带来更快的搜索,同时却增加创建索引的时间。因为对猪分片的修改,需要传递到更多的复制。</p>
<h2 id="sourceall">映射_source和_all</h2>
<p><a href="http://www.elasticsearch.org/guide/reference/mapping/">Mappings</a>定义了你的文档如何被索引和存储。你可以,比如说,定义每个字段的类型——比如你的syslog里,消息肯定是字符串,严重性可以是整数。怎么定义映射参见<a href="http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping.html">链接</a>。</p>
<p>映射有着合理的默认值,字段的类型会在新目录的第一条文档插入的时候被自动的检测出来。不过你或许会想自己来调控这点。比如,可能新目录的第一条记录的message字段里只有一个数字,于是被检测为长整型。当接下来99%的日志里肯定都是字符串型的,这样Elasticsearch就没法索引他们,只会记录一个错误日志说字段类型不对。这时候就需要显式的手动映射”message” : {“type” : “string”}。如何注册一个特殊的映射详见<a href="http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping.html">链接</a>。</p>
<p>当你使用基于时间的目录名时,在配置文件里创建索引模板可能更适合一点。详见<a href="http://www.elasticsearch.org/guide/reference/api/admin-indices-templates.html">链接</a>。除去你的映射,你海可以定义其他目录属性,比如分片数等等。</p>
<p>在映射中,你可以选择压缩文档的_source。这实际上就是整行日志——所以开启压缩可以减小索引大小,而且依赖你的设定,提高性能。经验值是当你被内存大小和磁盘速度限制的时候,压缩源文件可以明显提高速度,相反的,如果受限的是CPU计算能力就不行了。更多关于source字段的细节详见<a href="http://www.elasticsearch.org/guide/reference/mapping/source-field.html">链接</a>。</p>
<p>默认情况下,除了给你所有的字段分别创建索引,elasticsearch还会把他们一起放进一个叫_all的新字段里做索引。好处是你可以在_all里搜索那些你不在乎在哪个字段找到的东西。另一面是在创建索引和增大索引大小的时候会使用额外更多的CPU。所以如果你不用这个特性的话,关掉它。即使你用,最好也考虑一下定义清楚限定哪些字段包含进_all里。详见<a href="http://www.elasticsearch.org/guide/reference/mapping/all-field.html">链接</a>。</p>
<h2 id="section-5">刷新间隔</h2>
<p>在文档被索引后,Elasticsearch某种意义上是近乎实时的。在你搜索查找文档之前,索引必须被刷新。默认情况下,目录是每秒钟自动异步刷新的。</p>
<p>刷新是一个非常昂贵的操作,所以如果你稍微增大一些这个值,你会看到非常明显提高的插入速率。具体增大多少取决于你的用户可以接受到什么程度。</p>
<p>你可以在你的<a href="http://www.elasticsearch.org/guide/reference/api/admin-indices-templates.html">index template</a>里保存期望的刷新间隔值。或者保存在elasticsearch.yml配置文件里,或者通过(REST API)[http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings.html]升级索引设定。</p>
<p>另一个处理办法是禁用掉自动刷新,办法是设为-1。然后用<a href="http://www.elasticsearch.org/guide/reference/api/admin-indices-refresh.html">REST API</a>手动的刷新。当你要一口气插入海量日志的时候非常有效。不过通常情况下,你一般会采用的就是两个办法:在每次bulk插入后刷新或者在每次搜索前刷新。这都会推迟他们自己本身的操作响应。</p>
<h2 id="thrift">Thrift</h2>
<p>通常时,REST接口是通过HTTP协议的,不过你可以用更快的Thrift替代它。你需要安装<a href="https://github.com/elasticsearch/elasticsearch-transport-thrift">transport-thrift plugin</a>同时保证客户端支持这点。比如,如果你用的是<a href="https://github.com/aparo/pyes">pyes Python client</a>,只需要把连接端口从默认支持HTTP的9200改到默认支持Thrift的9500就好了。</p>
<h2 id="section-6">异步复制</h2>
<p>通常,一个索引操作会在所有分片(包括复制的)都完成对文档的索引后才返回。你可以通过<a href="http://www.elasticsearch.org/guide/reference/api/index_.html">index API</a>设置复制为异步的来让复制操作在后台运行。你可以直接使用这个API,也可以使用现成的客户端(比如pyes或者rsyslog的omelasticsearch),都会支持这个。</p>
<h2 id="section-7">用过滤器替代请求</h2>
<p>通常,当你搜索日志的时候,你感兴趣的是通过时间序列做排序而不是评分。这种使用场景下评分是很无关紧要的功能。所以用过滤器来查找日志比用请求更适宜。因为过滤器里不会执行评分而且可以被自动缓存。两者的更多细节参见<a href="http://www.elasticsearch.org/guide/reference/query-dsl/">链接</a>。</p>
<h2 id="section-8">批量索引</h2>
<p>建议使用<a href="http://www.elasticsearch.org/guide/reference/api/bulk.html">bulk API</a>来创建索引它比你一次给一条日志创建一次索引快多了。</p>
<p>主要要考虑两个事情:</p>
<ul>
<li>最佳的批量大小。它取决于很多你的设定。如果要说起始值的话,可以参考一下pyes里的默认值,即400。</li>
<li>给批量操作设定时器。如果你添加日志到缓冲,然后等待它的大小触发限制以启动批量插入,千万确定还要有一个超时限制作为大小限制的补充。否则,如果你的日志量不大的话,你可能看到从日志发布到出现在elasticsearch里有一个巨大的延时。</li>
</ul>
用systemtap定位nginx1.2在header解析时的报错
2012-08-23T00:00:00+08:00
monitor
nginx
systemtap
http://chenlinux.com/2012/08/23/systemtap-probe-nginx-http-header-parse-line
<p>一个url请求,在经过代理层访问应用层后,会报502错误。检查发现应用层是Nginx0.7.64+Resin3的结构,代理层是Nginx1.2。直接访问Nginx0.7.64是没问题的,访问Nginx1.2就会返回”upstream sent invalid header while reading response header from upstream”。</p>
<p>首先简单的通过编译–with-debug的nginx然后配置error_log debug;可以看到,nginx是在处理完Expires头后失败的。但是无法具体显示下一个头是在哪个地方。</p>
<p>所以进nginx/src/http/modules/ngx_http_proxy_module.c里,找到ngx_http_proxy_process_header函数,其中是根据ngx_http_parse_header_line函数的结果做判断的。所以出去看nginx/src/http/ngx_http_parse.c里ngx_http_parse_header_line函数,结果发现,从nginx1.0.14开始,新增加了关于空的判断:<br />
<code class="highlighter-rouge">c
if (ch == '\0') {
return NGX_HTTP_PARSE_INVALID_HEADER;
}
</code><br />
变更记录:<a href="http://trac.nginx.org/nginx/browser/nginx/tags/release-1.0.14/src/http">http://trac.nginx.org/nginx/browser/nginx/tags/release-1.0.14/src/http</a></p>
<p>说明如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>*) Headers with null character are now rejected.
Headers with NUL character aren't allowed by HTTP standard and may cause
various security problems. They are now unconditionally rejected.
</code></pre>
</div>
<p>但是这个NULL出现在哪里呢?这里就要用systemtap来查了~</p>
<p>安装不说了,直接yum或者apt-get都有。</p>
<p>介绍的话,有余锋大神的一系列slide。然后有官网的文档。另外发现了这个翻译beginner的<a href="http://blog.csdn.net/kafeiflynn/article/details/6429976">中文文档</a>。</p>
<p>最后在本例里的简单使用了:</p>
<p>首先要自己启动/usr/local/nginx/sbin/nginx程序;</p>
<p>然后运行systemtap命令:<br />
<code class="highlighter-rouge">bash
stap -e 'probe process("/usr/local/nginx/sbin/nginx").statement("ngx_http_parse_header_line@src/http/ngx_http_parse.c:855"){printf("%s\n",$$locals$$);}'
</code></p>
<p>最后发起出错的请求。查看stap的输出。类似下面这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>c='u' ch='U' p="ser-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5^M
Host: www.connect.renren.com^M
Pragma: no-cache^M
Accept: */*^M
Proxy-Connection: Keep-Alive^M
^M
e" hash=117 i=117 state=1 lowcase=""
</code></pre>
</div>
<p>说明:</p>
<ul>
<li>probe设探针</li>
<li>process运行命令</li>
<li>function指定函数</li>
<li>statement指定代码位置</li>
<li>$$vars变量</li>
<li>$$locals内部变量</li>
<li>$$parms参数变量</li>
</ul>
<p>上面三个变量后面加$显示具体内容或者成员</p>
<p>可以先printf(“%s\n”,$$locals)看到显示的是p的内存地址,每次++。然后$$locals$看具体内容。</p>
<p>最终观察到,在header中某行处理到第N个字节的时候,不再输出,即在该字节处碰到了NULL。</p>
Coro::Semaphore和async_pool示例
2012-08-20T00:00:00+08:00
testing
perl
http://chenlinux.com/2012/08/20/coro-async-pool-demo
<p>之前有一个<a href="http://chenlinux.com/2012/07/19/anyevent-fork-http-load-runner-demo/">AnyEvent和Fork写的http压测工具</a>,评论里有大神教导说用Coro控制并发更有效更方便。于是改写了下面的版本。从被压测的nginx server上,可以看到ESTABLISHED的数量确实大大增加,“”并发”两个字算是做到了。</p>
<p>先上普通的Coro::Semaphore控制并发的代码:<br />
<code class="highlighter-rouge">perl
# 之前的fork和count部分完全一致,不重复帖了。
sub coro_get {
my ($count, $urls) = @_;
my $data;
my @coros;
my $semaphore= Coro::Semaphore->new(1);
my $ua = FurlX::Coro->new();
for (1 .. $count) {
my $url = $urls->[int(rand($#{$urls}+1))];
push @coros, async {
my $guard = $semaphore->guard;
my $res = $ua->get($url);
$data->{'code'}->{$res->code}++;
};
}
$_->join for @coros;
return $data;
}
</code></p>
<p>然后贴的是用async_pool的代码,从perldoc来看据说是比async还快一倍。<br />
<code class="highlighter-rouge">perl
sub coro_pool_get {
my ($count, $urls) = @_;
my $sem = Coro::Semaphore->new( 1 - $Coro::POOL_SIZE );
my $ua = new FurlX::Coro;
my $data;
for( 1 .. $count ){
my $url = $urls->[int(rand($#{$urls}+1))];
async_pool { my $res = $ua->get("$url"); $data->{'code'}->{$res->status}++; $sem->up; };
};
$sem->down;
return $data;
};
</code></p>
<p>在两个函数中,只要修改$semaphore或者$limit的构造参数,就可以获得并发ESTABLISHED的效果。同样是4核nginx,基本在设置init var为100的情况下,最后整个脚本带来的是3000+的ESTABLISHED,然后可能出现少量的非200响应。</p>
一个Plack::Middleware的实例
2012-07-30T00:00:00+08:00
dancer
perl
http://chenlinux.com/2012/07/30/plack-middleware-demo
<p>做一个文件上传的网页,想稍微华丽一点,显示进度条出来。在Apache和Catalyst下都有现成的模块,不过Dancer上还没有。看了一下代码,Dancer::Request里没有像Catalyst那样暴露prepare_body_chunk方法。所以需要在plack上利用psgi.input来做。尽量重用现有代码,所以progress.css和progress.js都直接从Catalyst/Plugin/UploadProgress/example/Upload/root/static/里复制。<br />
```perl<br />
package Plack::Middleware::UploadProgress;<br />
use strict;<br />
use CHI;<br />
use Carp;<br />
use HTTP::Body;<br />
use Plack::Request;<br />
use Plack::TempBuffer;<br />
use parent qw(Plack::Middleware);</p>
<p>sub call {<br />
my ( $self, $env ) = @_;<br />
my $rm = $env->{REQUEST_METHOD};<br />
my $ct = $env->{CONTENT_TYPE};<br />
my $cl = $env->{CONTENT_LENGTH};<br />
if ( $rm =~ m#^post$#i and (<br />
$ct =~ m#^application/x-www-form-urlencoded#i or<br />
$ct =~ m#^multipart/form-data#i )<br />
) {<br />
my $cache = CHI->new( driver => ‘Memory’, global => 1 );<br />
my $req = Plack::Request->new($env);<br />
my $id = $req->param(‘progress_id’);</p>
<div class="highlighter-rouge"><pre class="highlight"><code> my $body = HTTP::Body->new($ct, $cl);
$env->{'plack.request.http.body'} = $body;
$body->cleanup(1);
my $input = $env->{'psgi.input'};
my $buffer;
if ($env->{'psgix.input.buffered'}) {
$input->seek(0, 0);
} else {
$buffer = Plack::TempBuffer->new($cl);
}
my $spin = 0;
while ($cl) {
$input->read(my $chunk, $cl < 8192 ? $cl : 8192);
my $read = length $chunk;
$cl -= $read;
$body->add($chunk);
$buffer->print($chunk) if $buffer;
my $progress = $cache->get('upload_progress_' . $id);
if ( !defined $progress ) {
$progress = {
size => $body->content_length,
received => length $chunk,
};
$cache->set( 'upload_progress_' . $id, $progress );
} else {
$progress->{received} += $read;
$cache->set( 'upload_progress_' . $id, $progress );
};
if ($read == 0 && $spin++ > 2000) {
Carp::croak "Bad Content-Length: maybe client disconnect? ($cl bytes remaining)";
}
};
if ($buffer) {
$self->env->{'psgix.input.buffered'} = 1;
$self->env->{'psgi.input'} = $buffer->rewind;
} else {
$input->seek(0, 0);
}
};
my $res = $self->app->($env);
return $res; }
</code></pre>
</div>
<p>true;<br />
```</p>
<p>上面的代码,progress部分基本摘抄自Catalyst::Plugin::UploadProgress模块,chunk部分基本摘抄自Plack::Middleware::CSRFBlock模块,当然它也基本摘抄自Plack::Request模块~~<br />
上面写的不全,没有关于GET /progress?progress_id=*的json输出处理。不过这部分也可以在DancerApp.pm里直接写get ‘/progress’ => sub {};,最后申明,这代码刚写完,没测……CHI或许应该用memcached而不是memory~</p>
<p>在DancerApp/bin/app.pl里这么配置:<br />
```perl<br />
use Dancer;<br />
use DancerApp;<br />
use Plack::Builder;</p>
<p>my $app = sub {<br />
my $env = shift;<br />
my $req = Dancer::Request-new( env => $env );<br />
};</p>
<p>builder {<br />
enable “Plack::Middleware::UploadProgress”;<br />
$app;<br />
};</p>
<p>dance;<br />
```</p>
一个Dancer::Plugin的实例
2012-07-30T00:00:00+08:00
dancer
perl
http://chenlinux.com/2012/07/30/dancer-plugin-demo
<p>公司内部工具统一使用passport认证登录,于是写这么一个小plugin,用来给dancer做的网站使用统一认证。<br />
passport的原理很简单,将原先的页面url带入session转到passport的login,然后由passport通过user/password或者kerberos确认是否正确,并返回一个ticket参数,然后拿这个ticket再到passport的verify上校验一次username,正确的话写入session即可。代码如下:<br />
```perl<br />
package Dancer::Plugin::Auth::Passport;<br />
use Dancer ‘:syntax’;<br />
use Dancer::Plugin;<br />
use Furl; <br />
use warnings;<br />
use strict;</p>
<p>our $VERSION = ‘0.01’;</p>
<p>my $settings = plugin_setting;<br />
my $PASSPORT_HOST = $settings->{passport_host} || ‘passport.company.com’;</p>
<p>sub _auth_passport {<br />
if ( !session(‘username’) ) {<br />
my $req_url = request->scheme .’://’. request->header(‘Host’) . request->uri;<br />
if ( defined(my $ticket = params->{‘ticket’}) ) {<br />
my $passport_url = “https://${PASSPORT_HOST}/verify.php?t=${ticket}”;<br />
my $furl = Furl->new( timeout => 10, headers => [‘Referer’ => “$req_url”] );<br />
my $res = $furl->get($passport_url);<br />
redirect “https://${PASSPORT_HOST}/login.php?forward=${req_url}” unless $res->is_success;<br />
session username => $res->content;<br />
redirect session->{‘original_url’};<br />
} else {<br />
session original_url => ${req_url};<br />
redirect “https://${PASSPORT_HOST}/login.php?forward=${req_url}”;<br />
};<br />
};<br />
};</p>
<p>register ‘auth_passport’ => \&_auth_passport;<br />
register_plugin;</p>
<p>true;<br />
<strong>END</strong><br />
```</p>
<p>使用方法如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">package</span> <span class="nv">DancerApp</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Dancer</span> <span class="s">':syntax'</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Dancer::Plugin::Auth::</span><span class="nv">Passport</span><span class="p">;</span>
<span class="nv">hook</span> <span class="s">'before'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="nv">auth_passport</span> <span class="p">};</span>
<span class="nv">get</span> <span class="s">'/'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="k">return</span> <span class="s">'Hello Passport'</span> <span class="p">};</span>
<span class="nv">true</span><span class="p">;</span>
</code></pre>
</div>
<p>完毕。</p>
用AnyEvent和ForkManager写一个http协议的压测工具
2012-07-19T00:00:00+08:00
testing
perl
http://chenlinux.com/2012/07/19/anyevent-fork-http-load-runner-demo
<p>话不多说,先上第一版的代码:<br />
```perl<br />
use Time::HiRes qw/time/;<br />
use AnyEvent::HTTP;<br />
use AnyEvent;<br />
use Coro;</p>
<p>my %code;<br />
my $count = $ARGV[0];<br />
my $url = “https://10.10.10.10/”;<br />
my $begin = time;<br />
my @coro = map {<br />
async {<br />
my $cv = AnyEvent::condvar;<br />
$cv->begin;<br />
my $header_time;<br />
http_request GET => “$url”,<br />
sub {<br />
my (undef, $hdr) = @<em>;<br />
$code{$hdr->{‘Status’}}++;<br />
$cv->end;<br />
}<br />
;<br />
$cv->recv;<br />
}<br />
} (1 .. $count);<br />
$</em>->join for @coro;<br />
print $cpus*$ARGV[0]/(time-$begin);<br />
```</p>
<p>上面这段脚本,作用是在每个进程中运行事件驱动的协程,以达到尽可能大的并发请求。</p>
<p>初步的测试,在单核Coro的情况下可以每秒发送1000+的https请求。</p>
<p>注意:如果使用的是AnyEvent::HTTP::LWP::UserAgent模块,虽然POD里写它用的其实就是AnyEvent::HTTP的代码套LWP的API格式,但实际只能用到30%的CPU,单核情况下的qps也就不到350的样子。</p>
<p>注意:本例测试的是HTTPS,AnyEvent::HTTP在TLS模式(即https请求)下,无法开启persistent连接。如果是普通http请求,开启persistent参数的qps应该会更高!</p>
<p>然后上第二版的代码,改用了EV循环,性能比Coro协程提高了大概5%的样子。使用了fork多进程,并且绑定到不同的CPU核上。<br />
```perl<br />
#!/usr/bin/perl<br />
use strict;<br />
use warnings;<br />
use autodie;<br />
use Parallel::ForkManager;<br />
use Time::HiRes qw/time/;<br />
use Sys::CpuAffinity;<br />
use AnyEvent::HTTP;<br />
use AE;<br />
use EV;</p>
<p>my $cpus = Sys::CpuAffinity::getNumCpus();<br />
my $pm = new Parallel::ForkManager($cpus, “/tmp/”);<br />
my @urls = qw(https://10.11.19.35/ https://10.11.21.121/);<br />
my $res;</p>
<p>$pm->run_on_finish( sub {<br />
my $data = $<em>[5];<br />
$res->{‘code’}->{$</em>} += $data->{‘code’}->{$_} for keys %{$data->{‘code’}};<br />
$res->{‘size’} += $data->{‘size’};<br />
$res->{‘time’} += $data->{‘time’};<br />
});</p>
<p>my $begin = time;<br />
foreach my $cpu (1 .. $cpus) {<br />
my $pid = $pm->start and next;<br />
Sys::CpuAffinity::setAffinity($$, 2 ** ($cpu-1));<br />
my $data = ae_get($ARGV[0], \@urls);<br />
$pm->finish(0, $data);<br />
}<br />
$pm->wait_all_children;<br />
my $use = time - $begin;</p>
<p>printf “%d fetches, %d max processes, in %.03f seconds\n”, $ARGV[0] * $cpus, $cpus, $use;<br />
printf “%.03f fetches/sec, %.03f bytes/sec\n”, $ARGV[0] * $cpus / $use, $res->{‘size’} / $res->{‘time’};<br />
printf “HTTP response codes:\n”;<br />
printf “ %d - %d\n”, $<em>, $res->{‘code’}->{$</em>} for sort keys %{$res->{‘code’}};</p>
<p>sub ae_get {<br />
my ($count, $urls) = @<em>;<br />
my $data;<br />
my $tmptime;<br />
my $cv = AE::cv;<br />
for (1 .. $count) {<br />
my $hdr_time;<br />
my $url = $urls->[int(rand($#{$urls}+1))];<br />
$cv->begin;<br />
http_request<br />
GET => “$url”, <br />
on_header => sub {<br />
$hdr_time = time;<br />
},<br />
sub {<br />
my (undef, $hdr) = @</em>;<br />
$data->{‘code’}->{$hdr->{‘Status’}}++;<br />
$data->{‘size’} += $hdr->{‘content-length’};<br />
$data->{‘time’} += ( time - $hdr_time );<br />
$cv->end;<br />
}<br />
;<br />
} <br />
$cv->recv;<br />
return $data;<br />
}<br />
```</p>
<p>增加了简单的统计功能,包括每秒请求数、平均下载速度,状态码汇总等。因为不好计算header的长度,所以只计算body部分的下载速度。<br />
增加了url列表功能,每次请求会随机的抽取其中的一个url。</p>
IPC::Locker模块介绍
2012-07-07T00:00:00+08:00
perl
http://chenlinux.com/2012/07/07/intro-ipc-locker
<p>当你需要给一个集群的某项服务做简单的排他性管理的时候,强力推荐Veripool公司的一系列模块:IPC::Locker、Schedule::Load。</p>
<p>今天先说IPC::Locker模块。部署很简单,直接在集群所有节点上运行cpanm IPC::Locker即可。该模块依赖几个都是perl的核心模块比如IO::Socket::INET、IO::Poll和POSIX。所以理论上你也可以把代码打个包分发。</p>
<p>随包分发的还有几个现成的脚本程序lockerd、lockersh、pidstat、pidstatd和pidwatch。</p>
<p>后面三个关注的remote设备上的pid是否存在等,但是相信一般情况下,我们不会自己来通过pid管理集群,所以在使用上只要理解lockerd和lockersh其实也是用pidstatd来解决pid问题的就够了。</p>
<p>其实代码很简单,看看就明白,无非就是lockerd用的IPC::Locker::Server是启动了一个IO::Socket::INET做tcp server,主要维护几个东西,一个是@{$self->{lock}}列表,一个是@{$self->{host}}列表,一个是$self->{locked}的Bool值。</p>
<p>而lockersh用的IPC::Locker则是连接上lockerd的端口,检查$self->{locked}状态,如果没locked就发送LOCK请求,然后fork一个进行exec你定义的shell命令,执行完成后,unlock发送UNLOCK请求给lockerd。</p>
<p>做个简单实验:</p>
<ol>
<li>在serverA上运行lockerd &</li>
<li>在serverB上运行lockersh –dhost serverA –lock test_task ‘while true;do echo “OK”;done’</li>
<li>在serverC上运行lockersh –dhost serverA –lock test_task ‘while true;do echo “OK”;done’</li>
<li>在serverD上运行lockersh –dhost serverA –lock other_task ‘while true;do echo “OK”;done’</li>
</ol>
<p>观察一下,结果是在serverB和serverD上同时在执行echo “OK”。而serverC被lock住了。继续:</p>
<ol>
<li>在serverB的session上按下Ctrl+C终止程序,然后再次运行上述命令</li>
<li>在serverC的session上按下Ctrl+C终止程序</li>
</ol>
<p>观察一下,结果是停止B时C的即开始,停止C的后B的继续。这些都不影响serverD的运行。</p>
<ol>
<li>终止serverD的程序,改为运行lockersh –dhost serverA –lock test_task ‘while true;do echo “OK”;done’</li>
</ol>
<p>观察一下,发现B、C、D是按照lockersh的执行次序解锁的。因为hostlist是一个列表,在server上是用for循环的。</p>
<p>注意:必须要先运行lockerd并且保证不中途退出。经过测试,如果lockerd中途退出再重新运行的话,因为locklist是保存在内存里会丢失的。结果就会出现之前的lockersh还在执行(他已经获得了lock,在unlock之前不会再和server通信的),之后再启动的新lockersh会在新lockerd上又获得一次lock的情况……</p>
<p>后一个Schedule::Load则可以根据集群设备的loadavg,top等,决定在哪台设备上运行job。还没测试。之后再记录。</p>
<p>补充:贴一个脚本,仿照lockersh改写的squid集群重启及报警控制:<br />
```perl<br />
#!/usr/bin/perl -w<br />
use FindBin;<br />
use lib “$FindBin::Bin/../lib”;</p>
<p>use strict;<br />
use warnings;<br />
use autodie;<br />
use vars qw ($Debug);<br />
use Furl;<br />
use IO::File;<br />
use Getopt::Long;</p>
<p>use IPC::Locker;<br />
use IPC::PidStat;</p>
<p>#======================================================================</p>
<p>my $pscount = <code class="highlighter-rouge">ps aux|grep -v grep|grep $0|wc -l</code>;<br />
print “Already run, waiting for lock now” and exit unless $pscount == 1;</p>
<p>#======================================================================</p>
<p>my %server_params = (lock=>[]);<br />
my $cluserv;</p>
<p>$Debug = 0;<br />
Getopt::Long::config (“require_order”);<br />
if (! GetOptions (<br />
“dhost=s” => sub {shift; $server_params{host} = shift;},<br />
“cluster=s” => sub {shift; push @{$server_params{lock}}, split(‘:’,shift);},<br />
“port=i” => sub {shift; $server_params{port} = shift;},<br />
“timeout=i” => sub {shift; $server_params{timeout} = shift;},<br />
“verbose!” => sub {shift; $server_params{verbose} = shift;},<br />
“debug” => \&debug,<br />
“service=s” => $cluserv,<br />
)) {<br />
die “%Error: Bad usage, see lockersh –help\n”;<br />
}</p>
<p>$#{$server_params{lock}}>=0 or die “%Error: –cluster not specified; see lockersh –help\n”;</p>
<h1 id="fork-once-to-start-parent-process">Fork once to start parent process</h1>
<p>my $foreground_pid = $$; # Unlike most forks, the job goes in the parent</p>
<h1 id="do-this-while-we-still-have-stderr">Do this while we still have STDERR.</h1>
<p>my $lock = new IPC::Locker (verbose=>0,<br />
timeout=>0,<br />
autounlock=>1,<br />
destroy_unlock=>0,<br />
%server_params,<br />
);<br />
$lock or die “%Error: Did not connect to lockerd,”;<br />
$lock->lock;</p>
<p>if (my $pid = fork()) { # Parent process, foreground job<br />
print “\tForeground: $cluserv\n” if $Debug;<br />
# The child forks again quickly. Sometimes, SIG_CHLD leaks to us and<br />
# wrecks the exec’d command, so wait for it now.<br />
my $rv = waitpid($pid, 0);<br />
if ($rv != $pid) {<br />
die “%Error: waitpid() returned $rv: $!”;<br />
} elsif ($?) {<br />
die “%Error: Child process died with status $?,”;<br />
}</p>
<div class="highlighter-rouge"><pre class="highlight"><code>print "Exec in $$\n" if $Debug;
&service($cluserv); } #else, rest is for child process.
</code></pre>
</div>
<h1 id="disassociate-from-controlling-terminal">Disassociate from controlling terminal</h1>
<p>POSIX::setsid() or die “%Error: Can’t start a new session: $!”;</p>
<h1 id="change-working-directory">Change working directory</h1>
<p>chdir “/”;<br />
open(STDIN, “+>/dev/null”) or die “%Error: Can’t re-open STDIN: $!”;<br />
if (!$Debug) {<br />
open(STDOUT, “+>&STDIN”);<br />
open(STDERR, “+>&STDIN”);<br />
}<br />
# Prevent possibility of acquiring a controlling terminal<br />
exit(0) if fork();</p>
<h1 id="wait-for-child-to-complete--we-cant-waitpid-as-were-not-the-parent">Wait for child to complete. We can’t waitpid, as we’re not the parent</h1>
<p>while (IPC::PidStat::local_pid_exists($foreground_pid)) { sleep 1; }<br />
print “Parent $foreground_pid completed\n” if $Debug;</p>
<h1 id="unlock">Unlock</h1>
<p>$lock->unlock; $lock=undef;<br />
print “Child exiting\n” if $Debug;</p>
<p>sub debug {<br />
$Debug = 1;<br />
$IPC::Locker::Debug = 1;<br />
}</p>
<p>sub service {<br />
my $cluserv = shift;<br />
die “Only support squid now!” unless $cluserv eq “squid”;<br />
die “Reload failed. Check squid.conf!” if eval “${cluserv}_reload”;<br />
while (1) {<br />
my $hit_rate = eval “${cluserv}_check”;<br />
notify “HIT Ratio: ${hit_rate}% now.\n”;<br />
exit if $hit_rate > 50;<br />
sleep 300;<br />
};<br />
}</p>
<p>sub squid_check {<br />
my $hit_rate;<br />
print “Run squid_check” if $Debug;<br />
my $squid_port = <code class="highlighter-rouge">awk '/^http_port/{print $2}' /etc/squid/squid.conf</code>;<br />
open my $fh, “squidclient -p ${squid_port} mgr:info |”;<br />
while (<$fh>) {<br />
next unless /^\s+Request Hit Ratios:\s+5min:\s*(-?\d+.\d)%,/;<br />
print “regex $1” if $Debug;<br />
$hit_rate = $1;<br />
last;<br />
}<br />
close $fh;<br />
return $hit_rate;<br />
}</p>
<p>sub squid_reload {<br />
print “Reload squid daemon. Do not reload within 10 mins of squid start” if $Debug;<br />
system(“squid”, “-k”, “reconfigure”);<br />
return $?;<br />
}</p>
<p>sub notify {<br />
my $furl = Furl->new(agent => “Clustrol/0.1”);<br />
$furl->post(“http://monitor.domain.com/eml/”,<br />
[ data => “$_” ],<br />
);<br />
}</p>
<p><strong>END</strong><br />
```</p>
AnyEvent::HTTPD和AnyEvent::HTTP使用实例
2012-07-02T00:00:00+08:00
perl
http://chenlinux.com/2012/07/02/anyevent-httpd-demo
<p>很简单的一个实例,就是开一个端口接受url请求,然后向squid提交这个url的刷新。<br />
```perl<br />
use AnyEvent::HTTPD;<br />
use AnyEvent::HTTP;</p>
<p>my $httpd = AnyEvent::HTTPD->new (port => 9090);<br />
my $ip = “127.0.0.1”;</p>
<p>$httpd->reg_cb (<br />
‘/’ => sub {<br />
my ($httpd, $req) = @<em>;<br />
my $urlpath = $req->url->path;<br />
http_request PURGE => “http://${ip}${urlpath}”, headers=> { “host”=>”host.domain.com”}, sub {<br />
my ($body, $hdr ) = @</em>;<br />
$req->respond([“$hdr->{‘Status’}”,”$hdr->{‘Reason’}”,{‘Content-Type’ => ‘text/html’}]);<br />
};<br />
},<br />
);</p>
<p>$httpd->run;</p>
<p>```</p>
<p>注意安装AnyEvent::HTTPD的时候,test需要Test::POD,但是Makefile.PL上没写,所以要先行安装。</p>
【翻译】Coro::Intro文档
2012-06-29T00:00:00+08:00
perl
http://chenlinux.com/2012/06/29/coro-intro
<!-- INDEX BEGIN -->
<div name="index">
<p><a name="__index__"></a></p>
<ul>
<li><a href="#coro简介">Coro简介</a></li>
<li><a href="#什么是coro_">什么是Coro?</a></li>
<li><a href="#协作线程">协作线程</a></li>
<ul>
<li><a href="#信号量和其他锁">信号量和其他锁</a></li>
<li><a href="#频道">频道</a></li>
<li><a href="#什么是我的_什么是我们的_">什么是我的,什么是我们的?</a></li>
<li><a href="#调试">调试</a></li>
</ul>
<li><a href="#真实世界里的事件循环">真实世界里的事件循环</a></li>
<ul>
<li><a href="#真实世界里的文件操作">真实世界里的文件操作</a></li>
<li><a href="#翻转控制____唤醒函数">翻转控制 —— 唤醒函数</a></li>
</ul>
<li><a href="#其他模块">其他模块</a></li>
<li><a href="#作者">作者</a></li>
</ul>
<hr name="index" />
</div>
<!-- INDEX END -->
<p>
</p>
<h1><a name="coro简介">Coro简介</a></h1>
<p>这个教程准备给你介绍Coro模块家族的最主要的几个特性。</p>
<p>本文首先介绍一些基础概念,然后简单的概述一下Coro家族的情况。</p>
<p>
</p>
<hr />
<h1><a name="什么是coro_">什么是Coro?</a></h1>
<p>Coro最早是作为一个协程的简单实现的模块开始的。它允许你捕获当前的运行点并且跳到另一个点去,又随时可以再跳回来。作为一种非局部的跳转,和C语言里的<code>setjmp</code>/<code>longjmp</code>没什么不同。这就是<a href="/Coro/State.html">the Coro::State manpage</a>模块。</p>
<p>这有一个很天然的应用场合,就是在协作线程中内置一个调度器和结果集,这也是当前Coro最主要的应用场景。很多文档和论文把这些“协作线程(cooperative threads)”叫做“协程(coroutines)”或者更简单的就写成Coros。</p>
<p>一个线程非常像一个精简的Perl解释器或者说进程:跟完整版的Perl解释器不同的地方就是线程并没有自己的局部变量或者代码的名字空间——这些都是共享的。这就意味着当一个线程修改了某个变量(包括通过引用修改任何值)时,其他线程如果使用同样的变量或者值时会立刻发现这个改变。</p>
<p>协作的意思,就是这些线程在涉及到CPU使用的时候必须相互配合——只有一个线程可以真正拥有CPU,如果有别的线程要用,当前运行的这个就要让出。后来的线程可以显式的调用函数来完成这个工作,也可以隐式的等待资源释放(比如信号量或者IO请求的完成)。这种线程模型在脚本语言(比如python或者ruby)中非常流行,而且我们这个实现比其他语言里的线程要高效的多。</p>
<p>Perl本身在这方面用词非常模糊——线程“thread”或者“ithread”实际上在别的地方又被叫做进程“process”:这个所谓的Perl线程实际上是在用于windows上的UNIX进程仿真代码。这就是为什么我们说他是进程而不是真的线程的原因。最大的区别就是,在进程和ithread线程之间,变量不是共享的。</p>
<p>
</p>
<hr />
<h1><a name="协作线程">协作线程</a></h1>
<p>Coro模块带给大家协作线程。首先你要<code>use</code>它:</p>
<p><code class="highlighter-rouge">perl
use Coro;</code></p>
<p>然后要创建线程,你可以使用Coro模块自动导出的<code>async</code>函数:</p>
<p><code class="highlighter-rouge">perl
async {print "hello\n";};</code></p>
<p>async期望的第一个参数是一个代码块(间接的对象符号)。你也可以传递更多的变量,他们会在执行的时候作为<code>@_</code>数组传递到代码块里面。不过因为是闭包的原因,你可能只需要引用当前可见的任何词法变量都行。</p>
<p>上面那行就已经创建了一个线程,但是如果你保存这行代码到文件里运行,你会发现自己看不到任何输出。</p>
<p>原因就是:虽然你已经创建了线程,这个线程也已经准备好了执行(<code>async</code>会加入到一个所谓的<em>ready queue</em>里),它却没有得到CPU时间来实际运行代码,因为main函数——实际也是一个一样的线程——一直霸占着CPU直到整个程序运行到结束。所以Coror的线程是协作的,main也要协作起来,要让出CPU来。</p>
<p>要显式的让出CPU,使用<code>cede</code>函数(在其他线程实现里经常被叫做<code>yield</code>):</p>
<p><code class="highlighter-rouge">perl
use Coro;
async {
print "hello\n";
};
cede;</code></p>
<p>运行上面的代码会打印出<code>hello</code>单词然后退出。</p>
<p>看起来不是很有趣,那让我们搞点稍微有趣的程序:</p>
<p><code class="highlighter-rouge">perl
use Coro;
async {
print "async 1\n";
cede;
print "async 2\n";
};
print "main 1\n";
cede;
print "main 2\n";
cede;</code></p>
<p>运行这个程序会打印出如下结果:</p>
<p><code class="highlighter-rouge">perl
main 1
async 1
main 2
async 2</code></p>
<p>这个例子很好的说明了它的非局部的跳跃能力:main先打印了第一回,然后释放CPU给其他线程。嗯,确实有其他线程,于是运行并打印“async 1”,然后这个线程也释放掉CPU。这时候只剩下一个线程就是main了,main于是接着运行。</p>
<p>让我们注意这个例子的更多细节部分:<code>async</code>创建一个新线程。所有的新线程开始都处于暂停状态。要运行的话这些线程就需要被放进ready队列,这是<code>async</code>做的第二件事。每次一个线程让出CPU的时候,Coro就会运行一个所谓的调度器<em>scheduler</em>,调度器选择ready队列中的下一个线程,把它从队列里挪出来运行。</p>
<p><code>cede</code>也做两件事情:第一它把一个运行中的线程放进ready队列里;然后它跳转到调度器。这实际上就是让出CPU。不过最终确保了线程被再次运行。</p>
<p>事实上,<code>cede</code>可以这样实现:</p>
<p><code class="highlighter-rouge">perl
sub my_cede {
$Coro::current->ready;
schedule;
}</code></p>
<p>这里<code>$Coro::current</code>永远都是包含了当前正在运行的线程,而<code>Coro::schedule</code>则是调度器的调用方法。</p>
<p>那如果不把当前线程放进ready队列里就先调用<code>schedule</code>的效果会怎样呢?很简单,调度器就自动找ready队列里下一个队列。而当前队列因为没放进ready队列里,就会一直沉睡直到有别的因素唤醒它。</p>
<p>下面这个例子,把当前线程记录在一个变量里,创建新线程,这样main线程就沉睡过去了。</p>
<p>然后新创建的线程使用rand来决定是否唤醒main线程,用的是之前变量的<code>ready</code>方法。</p>
<p><code class="highlighter-rouge">perl
use Coro;
my $wakeme = $Coro::current;
async {
$wakeme->ready if 0.5 > rand;
};
schedule;</code></p>
<p>现在,你运行这个程序,可能会发生两种情况:<code>async</code>线程唤醒了main,程序正常退出;或者没有唤醒main,得到的是如下提示:</p>
<p><code class="highlighter-rouge">perl
FATAL: deadlock detected.
PID SC RSS USES Description Where
31976480 -C 19k 0 [main::] [program:9]
32223768 UC 12k 1 [Coro.pm:691]
32225088 -- 2068 1 [coro manager] [Coro.pm:691]
32225184 N- 216 0 [unblock_sub scheduler] -</code></p>
<p>为什么会这样?嗯,当<code>async</code>线程执行到代码块的最后的时候,他就终止了(通过调用<code>Coro::terminate</code>),然后重新调用调度器。而之前<code>async</code>线程并没有唤醒main线程,ready队列里没有任何线程可用,程序无法继续了。所以当这里明明有线程<em>可以</em>运行(main)却没有<em>ready</em>,Coro最终得到了一个<em>死锁</em>信号——通常这时候你会看到一个所有线程的列表来帮你追踪问题。</p>
<p>然而现在有个非常重要的场景,<em>就是</em>事实上可能确实没有线程是ready的,但在一个事件驱动的程序里,程序依然可以前进。在这种程序里,某些线程肯尼个在等待一个外部事件,比如超时,比如通过socket到达的数据流。</p>
<p>这种场景下,死锁就不是很有用了。这下有个模块叫<a href="/Coro/AnyEvent.html">the Coro::AnyEvent manpage</a>用来集成线程到事件循环里。它配置Coro使得在这种情况下coro并不返回一个错误信息然后<code>die</code>掉,而是继续运行一个事件循环以期待收到哪个事件可以唤醒某些线程。</p>
<p>
</p>
<h2><a name="信号量和其他锁">信号量和其他锁</a></h2>
<p>仅仅依靠<code>ready</code>、<code>cede</code>和<code>schedule</code>来同步线程是非常困难的。尤其是如果同时有很多线程是ready状态的时候。Coro支持一些原语来帮助你更简单的同步线程。第一个就是<a href="/Coro/Semaphore.html">the Coro::Semaphore manpage</a>模块,它实现了信号量计数(二进制的信号量则是<a href="/Coro/Signal.html">the Coro::Signal manpage</a>模块,同样的还有<a href="/Coro/SemaphoreSet.html">the Coro::SemaphoreSet manpage</a>和<a href="/Coro/RWLock.html">the Coro::RWLock manpage</a>模块)。</p>
<p>信号量计数,某种意义上就是存储一个资源的计数。你可以通过调用<code>->down</code>方法来删除、分配、预留一个资源,这个方法会减去一个计数;同样调用<code>->add</code>方法可以添加或释放一个资源,这又增加一个计数。如果计数器值为<code>0</code>,<code>->down</code>方法就没法再减——也就是说被锁住了——线程就必须等待到计数器重新可用为止。</p>
<p>下面是例子:</p>
<p><code class="highlighter-rouge">perl
use Coro;
my $sem = new Coro::Semaphore 0; #初始化的信号是锁住的
async {
print "unlocking semaphore\n";
$sem->up;
};
print "trying to lock semaphore\n";
$sem->down;
print "we got it!\n";</code></p>
<p>这个程序创建一个<em>锁住</em>的信号(计数器为<code>0</code>)并且尝试锁住他(通过<code>down</code>方法减计数)。因为信号量已经耗尽,main线程会被阻塞住直到信号量恢复可用。</p>
<p>这样CPU就被转给了其他可读的线程,这里是用<code>async</code>创建的那个解锁信号量的线程(并且随即就终止了自己)。</p>
<p>既然信号量恢复了,main也就锁住他然后继续执行打印“we got it!”。</p>
<p>信号量计数最常用的地方是锁资源,或者说在使用和访问某个资源时排他。比如,假设有一个很耗内存的函数。你不想让多个线程同时调用这个函数,你可以这样写:</p>
<p><code class="highlighter-rouge">perl
my $lock = new Coro::Semaphore; #初始化未锁,默认是1
sub costly_function {
$lock->down; #引入锁
#进行其他操作
$lock->up; #解锁
}</code></p>
<p>不管有多少线程调用<code>costly_function</code>,只有一个可以运行他的代码块,其他的都在<code>down</code>调用时阻塞。如果你想限定的并发执行是5个,那就创建信号量的时候指定初始值为<code>5</code>.</p>
<p>为什么提到“操作块”?再次强调,Coro的线程是协作的:<code>costly_function</code>不释放CPU,所有的线程都不会运行。如果函数一直不释放,就显得锁有点多余了,不过在和外面的世界打交道的时候,这种情况太罕见了。</p>
<p>现在想想如果代码在<code>down</code>后,<code>up</code>前就<code>die</code>掉了。这导致信号量保持在一个锁的状态,这应该不会是你想要的——所以如果可能失败的地方,都把调用用<code>eval {}</code>包起来。</p>
<p>所以通常你希望在不管是正常还是异常的时候都释放锁的话,这里有个guard方法可能比较有用:</p>
<p><code class="highlighter-rouge">perl
my $lock = new Coro::Semaphore; #初始化时未锁定
sub costly_function {
my $guard = $lock->guard; # 获取监视
... # 开始做需要阻塞的动作
}</code></p>
<p>这个<code>guard</code>方法<code>down</code>掉信号量并返回一个所谓的guard对象。看起来这个对象除了有个引用外啥都不干,不过当所有的引用都完成,比如<code>costly_function</code>返回或抛出异常,它会自动的调用<code>up</code>恢复信号量,绝对不会忘掉滴。哪怕线程收到别的线程发来的<code>cancel</code>命令。</p>
<p>信号量和锁的介绍到此结束。除了<a href="/Coro/Semaphore.html">the Coro::Semaphore manpage</a>和<a href="/Coro/Signal.html">the Coro::Signal manpage</a>,还有读写锁的<a href="/Coro/RWLock.html">the Coro::RWLock manpage</a>和信号集<a href="/Coro/SemaphoreSet.html">the Coro::SemaphoreSet manpage</a>。他们都有自己的文档可查。</p>
<p>
</p>
<h2><a name="频道">频道</a></h2>
<p>信号量很不错,但通常你可能希望通过交换数据来进行通信。当然,你可以继续用锁、数组来通信,不过这里还有更有用的线程间通信抽象模块:<a href="/Coro/Channel.html">the Coro::Channel manpage</a>。频道是UNIX管道的Coro等价实现(也非常接近AmigaOS的消息端口)——你可以从一段放进去东西,然后从另一头读取出来。</p>
<p>下面是一个简单的例子,创建一个线程然后发送数字给它。然后这个线程计算这个数字的平方,通过另一个频道返回给main线程。</p>
<p><code class="highlighter-rouge">perl
use Coro;
my $calculate = new Coro::Channel;
my $result = new Coro::Channel;
async {
# 无限循环
while () {
my $num = $calculate->get; #获取数字
$num **= 2; #计算平方
$result->put ($num); #推进结果队列
}
};
for (1, 2, 5, 10, 77) {
$calculate->put ($_);
print "$_ ** 2 = ", $result->get, "\n";
}</code></p>
<p>得到结果是:</p>
<p><code class="highlighter-rouge">perl
1 ** 2 = 1
2 ** 2 = 4
5 ** 2 = 25
10 ** 2 = 100
77 ** 2 = 5929</code></p>
<p>这里面<code>get</code>和<code>put</code>方法都会阻塞当前线程:<code>get</code>首先检查是否<em>有</em>数据可用,没有就阻塞到数据到达为止。<code>put</code>同样,在频道到“最大容量”的时候阻塞。你不可能存储超过这个特定值的项目,这个值可以再创建频道的时候设置。</p>
<p>在上面的例子中,<code>put</code>不会阻塞,因为频道的默认容量是很高的。所以for循环首先put数据到频道里,然后开始试图<code>get</code>结果。这时候因为async线程还没有put东西出来(第一次迭代的时候他还没运行),result频道是空的,所以main线程在这里阻塞住了。</p>
<p>这时候唯一一个可运行的线程就是算平方的这个,于是它会被唤醒,<code>get</code>数据,然后计算平方,put到result频道,就此唤醒main线程,然后他继续运行,唤醒其他线程进入ready队列,就这样。</p>
<p>只有当async线程是从calculate频道<code>get</code>下一个数字的时候,他才会阻塞住(因为现在这个频道里没数据)然后main线程开始继续运行。依次类推。</p>
<p>这说明了Coro的一个总体原则:一个线程<em>只</em>在万不得已的时候才会阻塞。不管是Coro模块本身还是他的任一子模块,都是如此。因为他们在等待某些事件的发生。</p>
<p>不过小心了:当多个线程往<code>$calculate</code>放数据然后从<code>$result</code>里读出来的时候,他们可分不清楚谁是谁的。解决办法是用信号量,或者不单单发送数字,也发送自己专属的result频道。</p>
<p>
</p>
<h2><a name="什么是我的_什么是我们的_">什么是我的,什么是我们的?</a></h2>
<p>到底什么构成了线程?显然它包含有一个当前的执行点。不那么显然的,它还得有局部变量。是的,每个线程都要自己的一组局部变量。</p>
<p>想知道为什么这点是必须的么,看看下面这个例子吧:</p>
<p><code class="highlighter-rouge">perl
use Coro;
sub printit {
my ($string) = @_;
cede;
print $string;
}
async { printit "Hello, " };
async { printit "World!\n" };
cede; cede;</code></p>
<p>上面的代码最终打印的是<code>Hello, World!\n</code>。如果<code>printit</code>没有自己每个线程独立的<code>$string</code>变量,那打印的结果应该是<code>World!\nWorld!\n</code>。这绝对不是你想要的,而且会给线程的使用造成极大的麻烦。</p>
<p>为了让事情变的更顺利些,有不少东西都是线程独立的:</p>
<dl>
<dt><strong><a name="________和正则表达式的捕获变量________1__2等等" class="item">$_,@_,$@和正则表达式的捕获变量,$&,%+,$1,$2等等</a></strong></dt>
<dd>
<p><code>$_</code>用于局部变量,每个线程都是独立的(<code>$1</code>,<code>$2</code>之类的也一样);</p>
<p><code>@_</code>包括了参数,类似词法变量,也必须是线程独立的;</p>
<p><code>$@</code>不那么必须,但是独立的话会很好用。</p>
</dd>
<dt><strong><a name="__和默认的输出文件句柄" class="item">$/和默认的输出文件句柄</a></strong></dt>
<dd>
<p>线程在做IO的时候经常是阻塞的,而<code>$/</code>就是在读取每行的时候起作用,如果它是个共享变量,事情会很不方便。
默认输出文件句柄(参见<code>select</code>)的情况比较复杂:有时候全局的好,有时候线程独立的好。不过看起来后面这种情况更多一些,所以还是线程独立的了。</p>
</dd>
<dt><strong><a name="_sig___die___和_sig___warn___" class="item">$SIG{__DIE__}和$SIG{__WARN__}</a></strong></dt>
<dd>
<p>如果这两不是线程独立的话,下面这种常见的构造就没法协程切换了。</p>
```perl
eval {
local $SIG{__DIE__} = sub { ... };
...
};```
<p>既然异常处理是线程独立的,那么这些变量自然也需要如此了。</p>
</dd>
<dt><strong><a name="一些其他的深奥的玩意儿" class="item">一些其他的深奥的玩意儿</a></strong></dt>
<dd>
<p>比如说<code>$^H</code>变量就是线程独立的。很多类似这样额外的线程独立的东西不会直接被Perl访问,你通常不会注意到这些。</p>
</dd>
</dl>
<p>其他的东西都是线程间共享的。比如全局变量<code>$a</code>和<code>$b</code>。当你使用sort的时候,这两个变量变成特殊变量,然后如果你在排序的时候切换线程,或许结果会让你大吃一惊的。</p>
<p>另外一些<code>$!</code>,errno,<code>$.</code>,输入行号,<code>$,</code>,<code>$\</code>,<code>$"</code>和很多很多其他的特殊变量都是共享的。</p>
<p>虽然有些时候把他们局部化也不错,但一是他们用的不广泛,二是局部化的工作蛮困难的。</p>
<p>总之,如果未来发现哪个共享变量给Coro造成问题了,我们就可能把它改成线程独立的。</p>
<p>
</p>
<h2><a name="调试">调试</a></h2>
<p>有时候查出每个线程在做什么或者哪个线程出现在什么地方是蛮有用的。<a href="/Coro/Debug.html">the Coro::Debug manpage</a>模块就有这么一个方法,让你打印出一个和ps命令结果很像的列表——你可以在Coro检测到死锁前就查看。</p>
<p>使用方法如下:</p>
<p><code class="highlighter-rouge">perl
use Coro::Debug;
Coro::Debug::command "ps";</code></p>
<p>还记得上面求平方的例子吧?在<code>$calculate->get</code>后面运行ps方法,然后就会输出类似这样的结果:</p>
<pre>
PID SC RSS USES Description Where
8917312 -C 22k 0 [main::] [introscript:20]
8964448 N- 152 0 [coro manager] -
8964520 N- 152 0 [unblock_sub scheduler] -
8591752 UC 152 1 [introscript:12]
11546944 N- 152 0 [EV idle process] -
</pre>
<p>有趣的是后台运行的线程比我们想象中的要多。除掉这些额外的线程,main线程的pid是<code>8917312</code>,而<code>async</code>启动的线程的pid是<code>8591752.</code></p>
<p>后者也是唯一一个没有描述的线程,因为我们没有设置这个。设置方法就是<code>$Coro::current->{desc}</code>;</p>
<p><code class="highlighter-rouge">perl
async {
$Coro::current->{desc} = "cruncher";
...
};</code></p>
<p>在调试程序或者使用<a href="/Coro/Debug.html">the Coro::Debug manpage</a>的交互式shell的时候这个可能比较有用。</p>
<p>
</p>
<hr />
<h1><a name="真实世界里的事件循环">真实世界里的事件循环</a></h1>
<p>Coro强烈希望运行在一个事件驱动的程序里。事实上真实情况的Coro程序都是结合事件驱动技术或者多线程技术的。利用Coro也很方便就在这两个世界里做到很好的效果。</p>
<p>Coro可以通过<em>AnyEvent</em>模块(查看<a href="/Coro/AnyEvent.html">the Coro::AnyEvent manpage</a>的更多细节)自动集成到任何事件循环里,也可以接受<em>EV</em>和<em>Event</em>模块的特殊方法。</p>
<p>下面是一个简单的finger客户端,可以使用任何<em>AnyEvent</em>的事件循环:</p>
<p><code class="highlighter-rouge">perl
use Coro;
use Coro::Socket;
sub finger {
my ($user, $host) = @_;
my $fh = new Coro::Socket PeerHost => $host, PeerPort => "finger"
or die "$user\@$host: $!";
print $fh "$user\n";
print "$user\@$host: $_" while &lt;$fh>;
print "$user\@$host: done\n";
}
#验证几个账号
for (
(async { finger "abc", "cornell.edu" }),
(async { finger "sebbo", "world.std.com" }),
(async { finger "trouble", "noc.dfn.de" }),
) {
$_->join; #等待结果
}</code></p>
<p>这里又有些新东西。首先是<a href="/Coro/Socket.html">the Coro::Socket manpage</a>。这个模块的工作方式和<a href="/IO/Socket/INET.html">the IO::Socket::INET manpage</a>一样,除了它是协程的。也就是说,<a href="/IO/Socket/INET.html">the IO::Socket::INET manpage</a>在等待网络的时候会阻塞整个进程——就是说说所有线程都被阻塞了,这显然是不可取的。</p>
<p>另一方面,<a href="/Coro/Socket.html">the Coro::Socket manpage</a>却知道在等待网络的时候让出CPU给其他线程。这使得并发执行变得可能了。</p>
<p>另一个新东西是<code>join</code>方法:在这个例子里我们想要的就是启动三个<code>async</code>线程然后完成工作后退出。这可以用信号量计数,但是直接同步等待他们<code>terminate</code>更简单一些,这正是<code>join</code>方法做的。</p>
<p>无所谓三个<code>async</code>是不是按照他们<code>join</code>的顺序结束的——当线程还在运行的时候,join单纯就是等待。如果线程终止,他就获取返回值。</p>
<p>如果你之前有事件驱动编程的经验,你会发现上面的程序不太遵循常规的模式,也就是开始一些工作,然后运行事件驱动比如<code>EV::loop</code>。</p>
<p>事实上,重要程序都遵从这个模式,使用Coro也一样,所以和EV一起时Coro程序看起来是这样的:</p>
<p><code class="highlighter-rouge">perl
use EV;
use Coro;
#开始协程或者事件句柄
EV::loop; #然后循环</code></p>
<p>还有,为了调试,经常写成这样:</p>
<p><code class="highlighter-rouge">perl
use EV;
use Coro::Debug;
my $shell = new_unix_server Coro::Debug "/tmp/myshell";
EV::loop; #循环</code></p>
<p>这个程序在运行的同时会在UNIX套接字<em class="file">/tmp/myshell</em>上创建一个交互式shell。你可以用<em class="file">socat</em>程序访问它:</p>
<pre>
# socat readline /tmp/myshell
coro debug session. use help for more info
> ps
PID SC RSS USES Description Where
136672312 RC 19k 177k [main::] [myprog:28]
136710424 -- 1268 48 [coro manager] [Coro.pm:349]
> help
ps [w|v] show the list of all coroutines (wide, verbose)
bt <pid> show a full backtrace of coroutine <pid>
eval <pid> <perl> evaluate <perl> expression in context of <pid>
trace <pid> enable tracing for this coroutine
untrace <pid> disable tracing for this coroutine
kill <pid> <reason> throws the given <reason> string in <pid>
cancel <pid> cancels this coroutine
ready <pid> force <pid> into the ready queue
<anything else> evaluate as perl and print results
<anything else> & same as above, but evaluate asynchronously
you can use (find_coro <pid>) in perl expressions
to find the coro with the given pid, e.g.
(find_coro 9768720)->ready
loglevel <int> enable logging for messages of level <int> and lower
exit end this session</pre>
<p>好吧,微软用户可以使用<code>new_tcp_server</code>构造器。</p>
<p>
</p>
<h2><a name="真实世界里的文件操作">真实世界里的文件操作</a></h2>
<p>磁盘IO一般比网络IO快很多,但可能占用很长时间,这期间CPU本可以做其他的事情,现在却只能做一样。</p>
<p>幸运的是,CPAN上的<a href="/IO/AIO.html">the IO::AIO manpage</a>模块允许你把这些IO调用移到后台,而在前台做更有用的工作。这是基于事件/回调的,不过Coro很好的包装了它,叫做<a href="/Coro/AIO.html">the Coro::AIO manpage</a>模块,你可以在线程里很自然的使用它的函数:</p>
<p><code class="highlighter-rouge">perl
use Fcntl;
use Coro::AIO;
my $fh = aio_open "$filename~", O_WRONLY | O_CREAT, 0600
or die "$filename~: $!";
aio_write $fh, 0, (length $data), $data, 0;
aio_fsync $fh;
aio_close $fh;
aio_rename "$filename~", "$filename";</code></p>
<p>上面创建一个新文件,写入数据,同步到磁盘,然后自动的改成新的副本。</p>
<p>
</p>
<h2><a name="翻转控制____唤醒函数">翻转控制 —— 唤醒函数</a></h2>
<p>最后我说说翻转控制。这个控制指谁通知谁,谁在程序的控制内。在这个程序中,main程序就在控制中,并且传递这个控制给他调用的所有函数:</p>
<p><code class="highlighter-rouge">perl
use LWP;
#转移控制给get
my $res = get "http://example.org/";
#控制权返回给我们了
print $res;</code></p>
<p>当你切换到事件驱动程序的时候,不再是“我调用它”,“他调用我”这样——而是标题所说的翻转控制:</p>
<p><code class="highlighter-rouge">perl
use AnyEvent::HTTP;
#不用交出控制权太久,http_get立刻返回了
http_get "http://example.org/", sub {
print $_[0];
};
#我们继续拥有控制权并且可以做其他事情了</code></p>
<p>基于事件的编程很好,不过有时间它只是更简单的码字罢了,因为不用回调可以写得很像线性的样式。Coro也提供了一些特殊的函数来减少敲键盘的功夫:</p>
<p><code class="highlighter-rouge">perl
use AnyEvent::HTTP;
#不用交出控制权太久,http_get立刻返回了
http_get "http://example.org/", Coro::rouse_cb;
#我们继续拥有控制权并且可以做其他事情了
#相当于等待
my ($res) = Coro::rouse_wait;</code></p>
<p><code>Coro::rouse_cb</code>创建并返回一个特殊的回调。你可以把它传递给任意希望有回调的函数。</p>
<p><code>Coro::rouse_wait</code>等待(阻塞当前线程)最近创建的回调被调用,然后返回传给它的所有数据。</p>
<p>这两个函数允许你<em>机械的</em>翻转控制,由绝大多数基于事件的库使用的"基于回调"的样式变成"阻塞式"的样子,绝对如你所愿。</p>
<p>范例很简单,原先这样写:</p>
<p><code class="highlighter-rouge">perl
some_func ..., sub {
my @res = @_;
...
};</code></p>
<p>现在这样写:</p>
<p><code class="highlighter-rouge">perl
some_func ..., Coro::rouse_cb;
my @res = Coro::rouse_wait;
...</code></p>
<p>基于回调的接口很丰富,而这个唤醒函数允许你用一种更方便的方式来使用它们。</p>
<p>
</p>
<hr />
<h1><a name="其他模块">其他模块</a></h1>
<p>这篇介绍里只是提到了很少的几个方法和模块。Coro有很多其他的函数(参见<em>Coro</em>的文档)和模块(在<em>Coro</em>文档的<code>SEE ALSO</code>区域)。</p>
<p>值得注意的有<a href="/Coro/LWP.html">the Coro::LWP manpage</a> (并发LWP请求,不过单纯论HTTP的话,<a href="/AnyEvent/HTTP.html">the AnyEvent::HTTP manpage</a>是更好的替代选择),<a href="/Coro/BDB.html">the Coro::BDB manpage</a>,当你需要异步数据库的时候可用,<a href="/Coro/Handle.html">the Coro::Handle manpage</a>,当你需要在协程中使用文件句柄(通常访问<code>STDIN</code>和<code>STDOUT</code>)和<a href="/Coro/EV.html">the Coro::EV manpage</a>,优化的<em>EV</em>接口(<a href="/Coro/AnyEvent.html">the Coro::AnyEvent manpage</a>自动使用这个)。</p>
<p>有很多Coro相关的模块(参见i<a href="http://search.cpan.org/search?query=Coro&mode=module">http://search.cpan.org/search</a>)可能对解决你的问题有帮助。而且因为Coro和AnyEvent结合的很好,你也很容易就可以适应现有的AnyEvent模块(参见<a href="http://search.cpan.org/search?query=AnyEvent&mode=module">http://search.cpan.org/search</a>)。</p>
<p>
</p>
<hr />
<h1><a name="作者">作者</a></h1>
<pre>
Marc Lehmann <schmorp@schmorp.de>
<a href="http://home.schmorp.de/">http://home.schmorp.de/</a>
</pre>
STF介绍
2012-06-14T00:00:00+08:00
perl
http://chenlinux.com/2012/06/14/intro-stf
<p>STF项目,全称”<a href="http://en.wikipedia.org/wiki/Professional_wrestling_holds#STF">Stepover Toehold Facelock</a>“,原因是项目发起人喜欢这个动作,我勒个去……当然作者也给它找了个靠谱一点的解释,叫STorage Farm。</p>
<p>目前主要是日本三大门户之一<a href="http://blog.livedoor.com/">livedoor</a>和<a href="http://tou.ch/">loctouch</a>在使用。livedoor称其图片集群规模在70TB,400,000,000个对象(1,300,000,000个复制份),高峰流量带宽400Mbps。按照这个数据计算,大概图片的平均大小是50KB的样子。</p>
<p>perl系原先已经有一个非常著名的分布式文件系统,叫MogileFS,作者说STF与MogileFS的不同时说到:</p>
<ol>
<li>
<p>STF整个是基于HTTP的,而且是PSGI的。这里我理解MogileFS内部是HTTP的,但是fs对外的api是非http的。而且因为时间较早的原因,mogile内部的http服务器是Danga的socket基础的perlbal,现在perl世界都转向使用psgi了。</p>
</li>
<li>
<p>代码简单。MogileFS有28000行代码,STF只有6000行。我觉得这里因为mogile全套除了metadata用了mysql之外,全都是perl实现。而STF中采用了Q4M、mysql、memcached、nginx/apache等多种外部组件,加上psgi本身也很省代码。</p>
</li>
</ol>
<p>关于实际工作流程,作者坦言和MogileFS基本一样。</p>
<ol>
<li>
<p>dispatcher,类似mogile里的tracker,主要配置内容就是数据库连接。前端还有个proxy,要点是处理X-Reproxy-URL这个HTTP的header。STF中使用apache+mod_reproxy或者nginx,mogile中使用perlbal。</p>
</li>
<li>
<p>job queue,STF中使用Q4M或者theSchwartz,mogile中使用gearmand。用来通知worker进行replica等。</p>
</li>
<li>
<p>MySQL,存储除了文件实际内容以外的所有数据,这里STF和mogile一致。</p>
</li>
<li>
<p>memcached,为了提高性能,给mysql做缓存的。这里mogile没有,不过很容易在调优时改造加入。</p>
</li>
<li>
<p>admin interface,mogile是cli端的,STF是psgi的web端。</p>
</li>
<li>
<p>worker,做数据的replica,delete等,从Q4M里取任务。这个STF和mogile类似。</p>
</li>
<li>
<p>Storage,支持CRUD即GET/PUT/DELETE的HTTP服务器即可。mogile里是mogstored,STF里是storage.psgi。同样都要在admin interface里添加管理。</p>
</li>
</ol>
<p>然后介绍一些概念:</p>
<ol>
<li>
<p>object,一个url对象,因为STF和mogile一样设计目的是小图片,所以一般来说不会有超过大小的分多块的文件(原文a piece of data),mogile里cli专门针对大于64M的文件要指定–largefile一样。</p>
</li>
<li>
<p>bucket,一个逻辑上的group。object必须存在于bucket里。这里stf和mogile有些类似又别扭的地方。mogile中,逻辑顺序是这样的:domain->class->keyvalue;stf中,逻辑顺序是这样的:bucket->object。所以一个完整的GET请求会看到object的url是两层目录的样子。</p>
</li>
<li>
<p>entity,也就是replica的份数。</p>
</li>
</ol>
<p>然后是CRUD的协议:</p>
<ol>
<li>
<p>创建bucket:PUT /bucket HTTP/1.0即可,成功创建返回状态码201,已存在返回204,url格式不对返回400,其他返回500。因为apache的mod_reproxy模块不支持chunk,所以使用HTTP/1.0协议,不清楚nginx的话,是否可以用HTTP/1.1,不过我记得有文章说在处理小图片的时候,其实HTTP/1.0比HTTP/1.1更好,因为浏览器可以开更多并发连接。</p>
</li>
<li>
<p>删除bucket:DELETE /bucket HTTP/1.0\r\n\X-STF-Recursive-Delete: 1\r\n\r\n即可。这个多余出来的header可以指定删除bucket里所有的文件,否则会只清楚bucket保留文件,但是还不清楚这种情况下能否访问到这些孤儿文件呢?</p>
</li>
<li>
<p>创建object:PUT /bucket/path/to/my.png HTTP/1.0…即可。有两个附加的header,一个叫X-STF-Replication-Count,一个叫X-STF-Consistency。前者在保存好第一份之后返回响应,然后通过Q4M让worker开始replica完指定的其他份数;后者则等到指定的份数都完成后才返回响应。</p>
</li>
<li>
<p>获取object:GET /bucket/path/to/my.png HTTP/1.0即可。这里可以使用If-Modified-Since,也可以只使用HEAD请求。如果bucket不存在,响应状态码是500。</p>
</li>
<li>
<p>修改object:POST /bucket/path/to/my.png HTTP/1.0即可。不清楚为什么会提供modify功能。一般分布式系统都是搞追加而已。还需要测试的一个是如果直接POST新object会如何?</p>
</li>
<li>
<p>重命名object: MOVE /bucket/path/to/my.png HTTP/1.0\r\nX-STF-Move-Destination: /newbucket/newpath/to/my.png即可。又是一个古怪的需求。另外不清楚这两个是真修改了呢,还是在mysql里修改标记了而已。</p>
</li>
</ol>
【Logstash系列】使用Redis并自定义Grok匹配
2012-06-13T00:00:00+08:00
logstash
redis
http://chenlinux.com/2012/06/13/use-redis-and-grok-pattern-for-logstash
<p>之前提到,用RabbitMQ作为消息队列。但是这个东西实在太过高精尖,不懂erlang不会调优的情况下,很容易挂掉——基本上我这里试验结果跑不了半小时日志传输就断了。所以改用简单易行的redis来干这个活。</p>
<p>之前的lib里,有inputs/redis.rb和outputs/redis.rb两个库,不过output有依赖,所以要先gem安装redis库,可以修改Gemfile,取消掉相关行的注释,搜redis即可。</p>
<p>然后修改agent.conf:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">input</span> <span class="p">{</span>
<span class="n">file</span> <span class="p">{</span>
<span class="n">type</span> <span class="o">=></span> <span class="s2">"nginx"</span>
<span class="n">path</span> <span class="o">=></span> <span class="p">[</span><span class="s2">"/var/log/nginx/access.log"</span> <span class="p">]</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">output</span> <span class="p">{</span>
<span class="n">redis</span> <span class="p">{</span>
<span class="n">host</span> <span class="o">=></span> <span class="s2">"MyHome-1.domain.com"</span>
<span class="n">data_type</span> <span class="o">=></span> <span class="s2">"channel"</span>
<span class="n">key</span> <span class="o">=></span> <span class="s2">"nginx"</span>
<span class="n">type</span> <span class="o">=></span> <span class="s2">"nginx"</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>启动方式还是一样。</p>
<p>接着修改server.conf:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">input</span> <span class="p">{</span>
<span class="n">redis</span> <span class="p">{</span>
<span class="n">host</span> <span class="o">=></span> <span class="s2">"MyHome-1.domain.com"</span>
<span class="n">data_type</span> <span class="o">=></span> <span class="s2">"channel"</span>
<span class="n">type</span> <span class="o">=></span> <span class="s2">"nginx"</span>
<span class="n">key</span> <span class="o">=></span> <span class="s2">"nginx"</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">filter</span> <span class="p">{</span>
<span class="n">grok</span> <span class="p">{</span>
<span class="n">type</span> <span class="o">=></span> <span class="s2">"nginx"</span>
<span class="n">pattern</span> <span class="o">=></span> <span class="s2">"%{NGINXACCESS}"</span>
<span class="n">patterns_dir</span> <span class="o">=></span> <span class="p">[</span><span class="s2">"/usr/local/logstash/etc/patterns"</span><span class="p">]</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">output</span> <span class="p">{</span>
<span class="n">elasticsearch</span> <span class="p">{</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>然后创建Grok的patterns目录,主要就是github上clone下来的那个咯~在目录下新建一个叫nginx的文件,内容如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="no">NGINXURI</span> <span class="sx">%{URIPATH}</span><span class="p">(</span><span class="sc">?:</span><span class="o">%</span><span class="p">{</span><span class="no">URIPARAM</span><span class="p">})</span><span class="o">*</span>
<span class="no">NGINXACCESS</span> <span class="p">\[</span><span class="sx">%{HTTPDATE}</span><span class="p">\]</span> <span class="o">%</span><span class="p">{</span><span class="no">NUMBER</span><span class="ss">:code</span><span class="p">}</span> <span class="o">%</span><span class="p">{</span><span class="no">IP</span><span class="ss">:client</span><span class="p">}</span> <span class="o">%</span><span class="p">{</span><span class="no">HOSTNAME</span><span class="p">}</span> <span class="o">%</span><span class="p">{</span><span class="no">WORD</span><span class="ss">:method</span><span class="p">}</span> <span class="o">%</span><span class="p">{</span><span class="no">NGINXURI</span><span class="ss">:req</span><span class="p">}</span> <span class="o">%</span><span class="p">{</span><span class="no">URIPROTO</span><span class="p">}</span><span class="o">/</span><span class="sx">%{NUMBER:version}</span> <span class="sx">%{IP:upstream}</span><span class="p">(</span><span class="ss">:%</span><span class="p">{</span><span class="no">POSINT</span><span class="ss">:port</span><span class="p">})?</span> <span class="sx">%{NUMBER:upstime}</span> <span class="sx">%{NUMBER:reqtime}</span> <span class="sx">%{NUMBER:size}</span> <span class="s2">"(%{URIPROTO}://%{HOST:referer}%{NGINXURI:referer}|-)"</span> <span class="o">%</span><span class="p">{</span><span class="no">QS</span><span class="ss">:useragent</span><span class="p">}</span> <span class="s2">"(%{IP:x_forwarder_for}|-)"</span>
</code></pre>
</div>
<p>Grok正则的编写,可以参考<a href="https://github.com/logstash/logstash/wiki/Testing-your-Grok-patterns-%28--logstash-1.1.0-and-above-%29">wiki</a>进行测试。</p>
<p>也可以不写配置文件,直接用–grok-patterns-path参数启动即可。</p>
<p>ps: 考察了一下statsd,发现它也要另存一份数据,放弃掉。转研究Kibana界面和Elasticsearch的分布式。</p>
MogileFS安装
2012-06-10T00:00:00+08:00
perl
http://chenlinux.com/2012/06/10/install-mogilefs
<p>纯属凑数的更新,写写安装过程而已。没有调优,没有测评,嗯……</p>
<ul>
<li>storage Node</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code>cpanm MogileFS::Utils MogileFS::Client
<span class="c"># 因为MogileFS::Server的test里会测试mysql、sqlite、pgsql的支持,用不着,直接强制安装就好了</span>
cpanm --force MogileFS::Server
mkdir /etc/mogilefs
cat > /etc/mogilefs/mogstored.conf <span class="sh"><<EOF
maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
docroot=/data/mogstore
EOF
</span><span class="c"># 不在同一分区的磁盘采用软连接方式建立伪DEV设备</span>
mkdir /data/mogstore/dev170
mkdir /data1/mogstore
ln -s /data1/mogstore /data/mogstore/dev171
</code></pre>
</div>
<ul>
<li>Tracker</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code>yum install -y mysql-server mysql-devel
cpanm DBI DBD::mysql MogileFS::Utils MogileFS::Client
cpanm --force MogileFS::Server
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">CREATE</span> <span class="k">DATABASE</span> <span class="n">MogileFS</span> <span class="k">DEFAULT</span> <span class="n">CHARACTER</span> <span class="k">SET</span> <span class="n">utf8</span> <span class="k">COLLATE</span> <span class="n">utf8_bin</span><span class="p">;</span>
<span class="k">grant</span> <span class="k">all</span> <span class="k">on</span> <span class="n">MogileFS</span><span class="p">.</span><span class="o">*</span> <span class="k">to</span> <span class="s1">'mogile'</span><span class="o">@</span><span class="s1">'%'</span> <span class="n">identified</span> <span class="k">by</span> <span class="s1">'mogile'</span><span class="p">;</span>
<span class="k">UPDATE</span> <span class="n">mysql</span><span class="p">.</span><span class="k">user</span> <span class="k">SET</span> <span class="n">Password</span><span class="o">=</span><span class="n">PASSWORD</span><span class="p">(</span><span class="s1">'newpass'</span><span class="p">)</span> <span class="k">WHERE</span> <span class="k">User</span><span class="o">=</span><span class="s1">'mogile'</span><span class="p">;</span>
<span class="n">FLUSH</span> <span class="k">PRIVILEGES</span><span class="p">;</span>
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># 初始化mysql数据库表</span>
mogdbsetup --dbhost<span class="o">=</span>tracker.mogile.com --dbname<span class="o">=</span>MogileFS --dbuser<span class="o">=</span>mogile --dbpass<span class="o">=</span>newpass
mkdir /etc/mogilefs
cat > /etc/mogilefs/mogilefsd.conf <span class="sh"><<EOF
db_dsn = DBI:mysql:MogileFS:host=tracker.mogile.com
db_user = mogile
db_pass = mogile
listen = 0.0.0.0:7001
conf_port = 7001
query_jobs = 10
delete_jobs = 1
replicate_jobs = 5
reaper_jobs = 1
EOF
</span><span class="c"># mogilefsd不能用root启动</span>
useradd mogile
su - mogile -c <span class="s1">'mogilefsd -c /etc/mogilefs/mogilefsd.conf --daemon'</span>
<span class="c"># 添加storage node和相关伪DEV设备信息</span>
mogadm host add mognode17 --ip<span class="o">=</span>10.0.0.17 --port<span class="o">=</span>7500 --status<span class="o">=</span>alive
mogadm device add mognode17 170
mogadm device add mognode17 171
<span class="c"># 添加域和类。文件的key在同一域内是唯一的。同一类可以指定自己的复制份数</span>
mogadm domain add fmn
mogadm class add fmn large --mindevcount<span class="o">=</span>3
mogadm class add fmn small --mindevcount<span class="o">=</span>3
<span class="c"># 检查状态,类似的有mogadm check命令</span>
mogstats --db_dsn<span class="o">=</span><span class="s2">"DBI:mysql:MogileFS:host=tracker.mogile.com"</span> --db_user<span class="o">=</span><span class="s2">"mogile"</span> --db_pass<span class="o">=</span><span class="s2">"mogile"</span> --verbose --stats<span class="o">=</span><span class="s2">"all"</span>
<span class="c"># 插入一个文件做测试</span>
mogtool --debug --trackers<span class="o">=</span>127.0.0.1:7001 --domain<span class="o">=</span>fmn --class<span class="o">=</span>large inject index.html <span class="s2">"index.html"</span>
</code></pre>
</div>
<ul>
<li>Fuse</li>
</ul>
<p>这步没搞出来,因为search.cpan.org上的Fuse是0.14版本,cpanm安装居然说无法下载的是0.15版。而MogileFS::Client::Fuse里use的是0.11版……<br />
而且直接下载的Fuse0.14版源码编译还一直通不过make test。。。</p>
<p>最后在github上找到两个资源:<br />
<a href="https://github.com/dpavlin/perl-fuse">Fuse-0.15</a> <br />
<a href="https://github.com/frett/MogileFS-Fuse">Mogile-Fuse-0.03</a></p>
<p>先解决依赖:<br />
<code class="highlighter-rouge">bash
yum install -y fuse fuse-devel fuse-libs
cpanm FUSE::Client FUSE::Server Lchown Filesys::Statvfs Unix::Mknod
</code><br />
然后perl Makefile.PL && make && make test && make install即可。<br />
挂载命令是:<br />
mount-mogilefs –daemon –tracker 10.0.0.16:7001 /mnt/mogilefs<br />
不过目前为止我挂载上去依然无法使用,输入输出有问题……</p>
淘宝TrafficServer的SSD分支测试与介绍
2012-06-08T00:00:00+08:00
CDN
ats
http://chenlinux.com/2012/06/08/intro-trafficserver-ssd-branch-in-taobao
<p>同事介绍说淘宝有关于trafficserver的一个分支支持SSD的。下来试试。<a href="http://gitorious.org/trafficserver/taobao/commits/tbtrunk_ssd">下载地址</a></p>
<p>官方的ats安装过程很简单,这个分支稍微麻烦一些,因为有些包变成强制依赖了,包括:expat/openssl/pcre等。根据提示安装好再重新编译即可。</p>
<p>表面看几乎没什么差别,就是多了一个storage_ssd.config配置文件。</p>
<p>基础配置项说明网上都有,无外乎就是records.config里的监听,remap.config里的域名,storage.config里的目录。下面说几个比较怪异或者难受的地方:</p>
<ol>
<li>
<p>trafficserver极其依赖DNS解析。我在parents.config中定义了多个parents的IP:port后,打开records.config里的debug信息,包括http.*、dns.*、cdn.*和url.*,结果发现ats针对每个url,会在获取完parents后依然去请求dns解析——注意这是在records.config里已经配置了no_dns_just_forward_to_parent为1之后,这种与字面意思严重不一样的结果让我很诧异……</p>
</li>
<li>
<p>由于上面说的parent配置不可行,所以在remap.config中只能使用单个ip的方式回源。这里配置说明一般都说前后域名必须不一样,但是我在debug中没有发现这个逻辑,事实上ats不管这个事情,所谓不能一样,大概是怕本机dns解析回到自己变成loop吧。</p>
</li>
<li>
<p>默认关于内存,是每1GB的磁盘启用1MB的内存。所以如果不指定的话,基本上磁盘的IO会很高很高!而最恶心的地方来了:配置里所有的数值都是以Byte为单位的,尼玛写4GB内存写的跟裹脚布一样长啊!</p>
</li>
</ol>
<p>然后说测试。在性能方面,ats还是不错的。基本上我在单机走lo口,用http_load做压测,都是http_load先到瓶颈——因为http_load只用单核CPU。使用-p 1000的参数测试(因为最大只让开到1020),同机上的squid2.6.23、trafficserver和trafficserver-ssd三种,rps分别在6000,19000和14000的样子。first-bytes msec在10,4,1的样子,cpu在150%,25%,25%的样子。</p>
<p>以上是只用了4个域名每个域名1个url的小样本测试。实际运行起来应该不会这么高。</p>
<p>另:在需要dns解析的时候,14000的rps降到只有300+,深表无语。</p>
<hr />
<p>线上流量运行测试后的补充:</p>
<ul>
<li>不要使用文件存储,直接把裸设备交给ats管理!</li>
<li>预估好要缓存的文件的平均大小,默认是8000。ats和squid不一样的是这个值会影响ats的文件组织结构。每次修改都会重建缓存。</li>
</ul>
用ganglia监控trafficserver
2012-06-08T00:00:00+08:00
monitor
ganglia
python
ats
http://chenlinux.com/2012/06/08/create-python-module-for-ganglia
<p>trafficserver提供了几种很不错的性能监控方式。首先是一个模仿cisco的shell工具./bin/traffic_shell——这个工具可以set变量,也可以show变量,另一个是类似squidclient的./bin/traffic_line工具——这个工具同样可以set和show变量,不过这里变量更接近源代码函数名的样子,相当于调用API了。此外还有Perl和Web的其他方式……</p>
<p>注:有些变量是可以动态修改的,有些比如内存大小之类的必须重启才生效。</p>
<p>用shell工具最方便,因为你单写个show,会提示你所有可以show的参数。一般性能方面就是http-stats/http-trans-stats/proxy-stats/cache-stats这几个。淘宝ssd分支在这里增加了一个SSD吞吐量和RAM缓存命中率的显示。不过一开始,你会发现RAM Cache HITs一直是0.00%!!询问作者后才知道,为了尽量提高性能,默认配置下RAM命中后不统计数据直接pass了……需要在records.config里配置CONFIG proxy.config.http.record_tcp_mem_hit = 1 才行——这配置records.config.default里还没有,呵呵~~</p>
<p>长期监控来说,用shell工具就不方便了,就要改用line工具,不过这里line读取的API,官方文档里是不全的,有一个简单的办法,就是进src/mgmt/cli/ShowCmd.cc里去把Cli_RecordGetInt里的字符串全grep出来,然后慢慢挑拣吧~~</p>
<p>下面是一个python的脚本,用来在ganglia里监控几个我个人认为比较重要的trafficserver性能参数的。<br />
```python<br />
import os<br />
import re<br />
import sys<br />
import time</p>
<p>ts_line = ‘/usr/local/trafficserver-ssd/bin/traffic_line’</p>
<p>def metric_read(name):<br />
command = “%s -r %s” % (ts_line, name)<br />
result = os.popen(command).readlines()<br />
if re.search(“ratio|percent”,name):<br />
return float(result[0]) * 100<br />
else:<br />
return float(result[0])</p>
<p>descriptors = [<br />
{<br />
‘description’:’Cache Bytes used’,<br />
‘name’: ‘proxy.process.cache.bytes_used’,<br />
‘call_back’: metric_read,<br />
‘time_max’:10,<br />
‘value_type’:’float’,<br />
‘units’:’Bytes’,<br />
‘slope’:’both’,<br />
‘format’:’%f’,<br />
‘groups’:’trafficserver’<br />
},<br />
{<br />
‘description’:’Cache size’,<br />
‘name’: ‘proxy.process.cache.bytes_total’,<br />
‘call_back’: metric_read,<br />
‘time_max’:10,<br />
‘value_type’:’float’,<br />
‘units’:’Bytes’,<br />
‘slope’:’both’,<br />
‘format’:’%f’,<br />
‘groups’:’trafficserver’<br />
},<br />
{<br />
‘description’:’Transactions per second’,<br />
‘name’: ‘proxy.node.user_agent_xacts_per_second’,<br />
‘call_back’: metric_read,<br />
‘time_max’:10,<br />
‘value_type’:’float’,<br />
‘units’:’requests/sec’,<br />
‘slope’:’both’,<br />
‘format’:’%f’,<br />
‘groups’:’trafficserver’<br />
},<br />
{<br />
‘description’:’Document hit rate’,<br />
‘name’: ‘proxy.node.cache_hit_ratio_avg_10s’,<br />
‘call_back’: metric_read,<br />
‘time_max’:10,<br />
‘value_type’:’float’,<br />
‘units’:’%’,<br />
‘slope’:’both’,<br />
‘format’:’%f’,<br />
‘groups’:’trafficserver’<br />
},<br />
{<br />
‘description’:’Bandwidth savings’,<br />
‘name’: ‘proxy.node.bandwidth_hit_ratio_avg_10s’,<br />
‘call_back’: metric_read,<br />
‘time_max’:10,<br />
‘value_type’:’float’,<br />
‘units’:’%’,<br />
‘slope’:’both’,<br />
‘format’:’%f’,<br />
‘groups’:’trafficserver’<br />
},<br />
{<br />
‘description’:’Cache percent free’,<br />
‘name’: ‘proxy.node.cache.percent_free’,<br />
‘call_back’: metric_read,<br />
‘time_max’:10,<br />
‘value_type’:’float’,<br />
‘units’:’%’,<br />
‘slope’:’both’,<br />
‘format’:’%f’,<br />
‘groups’:’trafficserver’<br />
},<br />
{<br />
‘description’:’Total document bytes from client’,<br />
‘name’: ‘proxy.process.http.user_agent_response_document_total_size’,<br />
‘call_back’: metric_read,<br />
‘time_max’:10,<br />
‘value_type’:’float’,<br />
‘units’:’Bytes’,<br />
‘slope’:’both’,<br />
‘format’:’%f’,<br />
‘groups’:’trafficserver’<br />
},<br />
{<br />
‘description’:’Total SSD Serve Bytes’,<br />
‘name’: ‘proxy.process.http.ssd_serve_total_size’,<br />
‘call_back’: metric_read,<br />
‘time_max’:10,<br />
‘value_type’:’float’,<br />
‘units’:’Bytes’,<br />
‘slope’:’both’,<br />
‘format’:’%f’,<br />
‘groups’:’trafficserver’<br />
},<br />
{<br />
‘description’:’RAM Cache Hits’,<br />
‘name’: ‘proxy.node.cache_hit_mem_ratio’,<br />
‘call_back’: metric_read,<br />
‘time_max’:10,<br />
‘value_type’:’float’,<br />
‘units’:’%’,<br />
‘slope’:’both’,<br />
‘format’:’%f’,<br />
‘groups’:’trafficserver’<br />
},<br />
{<br />
‘description’:’HTTP Transaction Fresh Speeds’,<br />
‘name’: ‘proxy.node.http.transaction_msec_avg_10s.hit_fresh’,<br />
‘call_back’: metric_read,<br />
‘time_max’:10,<br />
‘value_type’:’float’,<br />
‘units’:’ms’,<br />
‘slope’:’both’,<br />
‘format’:’%f’,<br />
‘groups’:’trafficserver’<br />
},<br />
{<br />
‘description’:’HTTP Transaction Now Cached Speeds’,<br />
‘name’: ‘proxy.node.http.transaction_msec_avg_10s.miss_cold’,<br />
‘call_back’: metric_read,<br />
‘time_max’:10,<br />
‘value_type’:’float’,<br />
‘units’:’ms’,<br />
‘slope’:’both’,<br />
‘format’:’%f’,<br />
‘groups’:’trafficserver’<br />
},<br />
]</p>
<p>def metric_init(params):<br />
return descriptors</p>
<p>def metric_cleanup():<br />
pass</p>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:<br />
metric_init(None)</p>
<div class="highlighter-rouge"><pre class="highlight"><code>while 1:
for d in descriptors:
v = d['call_back'](d['name'])
print '%s = %.2f' % (d['name'], v)
time.sleep(2)
print '-'*11
</code></pre>
</div>
<p>```<br />
个人之前从来没写过python,这个脚本完全照葫芦画瓢,万幸确实可以运行……</p>
<p>不过不太明白的就是,如果把def metric_read定义在descriptors数组后面,运行会报错。很奇怪python为什么会这样?</p>
【Logstash系列】用rabbitmq和elasticsearch搭建分布式日志收集存储系统
2012-06-01T00:00:00+08:00
logstash
rabbitmq
elasticsearch
http://chenlinux.com/2012/06/01/dist-logstash-and-elasticsearch
<p>上上篇讲到怎样用MRI的ruby在客户端收集日志。今天主要注意服务器端,考虑grok、elastic、web这几个功能在JRuby上才好。所以服务器端可以再开一个JRuby的进程。</p>
<ul>
<li>首先安装RabbitMQ的过程</li>
</ul>
<p>最简单的办法,采用epel的yum,或者apt安装。其次简单的办法,从rabbitmq上下载bin的tar.gz。 <br />
这里需要注意一下,rabbitmq-server启动的时候,默认node启动在rabbit@${hostname}上。而且这个hostname不是fqdn的,是第一个主机名。比方说你的hostname是MyHome-1.mydomain.com.,那node就是rabbit@MyHome-1。这个时候很容易报Connect MyHome-1 timeout。所以/etc/hosts一定要写好。 <br />
rabbitmq-server起来之后,可以用rabbitmyctl来具体的创建user啊,vhost啊之类的东西,作为测试,我们就直接使用默认的guest用户和/了。</p>
<ul>
<li>然后安装elasticsearch的过程</li>
</ul>
<p>这一步在logstash的docs里讲的很清楚了,就是下载tar.gz,解压然后java运行起来即可: <br />
<code class="highlighter-rouge">bash
ES_PACKAGE=elasticsearch-0.18.7.zip
ES_DIR=${ES_PACKAGE%%.zip}
SITE=https://github.com/downloads/elasticsearch/elasticsearch
if [ ! -d "$ES_DIR" ] ; then
wget --no-check-certificate $SITE/$ES_PACKAGE
unzip $ES_PACKAGE
fi
</code></p>
<ul>
<li>部署一个logstash的采集节点</li>
</ul>
<p>和上篇所述一样,传输一个删减版的Gemfile到采集节点。然后使用bundle安装这些模块: <br />
<code class="highlighter-rouge">bash
mkdir -p /usr/local/logstash/etc /usr/local/logstash/bin /usr/local/logstash/lib
scp ${logstashmaster}:/usr/local/logstash/Gemfile /usr/local/logstash/
scp -rf ${logstashmaster}:/usr/local/logstash/lib/* /usr/local/logstash/lib/
scp ${logstashmaster}:/usr/local/logstash/bin/logstash /usr/local/logstash/bin/
gem install bundler
cd /usr/local/logstash/
bundle install
</code><br />
然后编写一个使用rabbitmq的配置文件:<br />
```ruby<br />
input {<br />
file {<br />
type => “syslog”<br />
path => [“/var/log/syslog.log”, “/var/log/messages” ]<br />
} <br />
}</p>
<p>output {<br />
amqp {<br />
host => “MyHome-1”<br />
exchange_type => “fanout”<br />
name => “rawlogs”<br />
}<br />
}<br />
```<br />
OK,用ruby /usr/local/logstash/bin/logstash agent -f /usr/local/logstash/etc/agent.conf启动即可。</p>
<ul>
<li>部署一个logstash的汇聚节点</li>
</ul>
<p>这一步因为用到的模块大多是JRuby的,所以可以直接使用jar包的方式简单搞定。<br />
编写一个使用rabbitmq和elasticsearch的配置文件:<br />
```ruby<br />
input {<br />
amqp {<br />
type => “syslog”<br />
host => “MyHome-1”<br />
exchange => “rawlogs”<br />
name => “rawlogs_consumer”<br />
}<br />
}</p>
<p>filter {<br />
grok {<br />
type => “syslog”<br />
pattern => “%{SYSLOG}”<br />
}<br />
}<br />
output {<br />
elasticsearch { }</p>
<p>}<br />
```<br />
这里比较讨厌的还是rabbitmq的部分。假如前面的步骤rabbitmq-server压根启动失败了,这里amqp不会返回报错说连接失败或者连接node超时什么的,而是说你试图连接一个私有的被锁定的队列……</p>
<ul>
<li>部署一个logstash的展示节点</li>
</ul>
<p>这个节点就没必要再单开一台了,就用上面的jar包再启动一个web即可:java -jar logstash-1.1.0-monolithic.jar agent -f server.conf – web –backend ‘elasticsearch:///?local’</p>
<ul>
<li>测试</li>
</ul>
<p>现在可以打开浏览器访问web查看了。很简单的页面,顶上一个搜索栏,中间一个按时间轴显示的柱状图,下面是具体的日志记录。点具体的某条日志,会有浮框显示该条记录的详细信息(host/date/event/message等)</p>
<p>下一步研究grok正则匹配的编写,然后stated实时绘图,lucene查询语法。</p>
用Spork和Template::Toolkit生成slides胶片展示
2012-06-01T00:00:00+08:00
perl
http://chenlinux.com/2012/06/01/create-slides-by-using-spork-and-tt2
<p>在不少技术集会上,大家都不再采用ppt而是使用pdf,甚至浏览器,编辑器来显示内容。利用js和css完成slide效果已经越来越花哨。另一个用vim的流派也让人很是惊叹~ <br />
那不会css的童鞋怎么办呢?这里有个笨办法。用Spork工具生成html页——反正slide一般内容不多,html代码重复一点也不浪费啥事儿~用template技术刚刚好。<br />
Spork是一个在Sporx基础上构成的工具,可以直接cpan安装,不过默认情况下没有plugin。所以比较好的办法是上git找个带plugin的代码clone。这样页面样式好看点。 <br />
比方我用的这个就是Spork作者用的:<a href="https://github.com/ingydotnet/spork-pm">Spork-pm</a></p>
<p>新建一个slide的工程相当简单:<br />
<code class="highlighter-rouge">bash
git clone https://github.com/ingydotnet/spork-pm.git
cd spork-pm
perl Makefile.PL
make && make install
# 必须新建,非空目录时命令无效
mkdir /tmp/slides
cd /tmp/slides
Spork -new
Spork -make
Spork -start
</code></p>
<p>在slide里每张都加载,css和js都是重复的。make的过程就是使用template展开的问题。start就是打开浏览器的命令行的alias。这里其实可以优化一下改成外链css/js(默认是因为有些控制bgcolor啊之类的spork语法可以在单页里用,所以就没外链)~ <br />
注意template里的模板html里charset都是写的UTF-8编码,而在win上,编辑器默认是GB2312的,需要更改过来。</p>
<p>具体语法,—-表示一页;==表示大标题;<em>表示小条目,</em>越多条目等级越低;+表示稍后显示(其实就是在本页基础上再开新页)。 <br />
然后还有一些config定义,用来显示head,foot和start页内容的。</p>
<p>最后,上一个最近我的slide,关于DNS协议和应用的内容,浏览地址见:<br />
<a href="http://chenlinux.com/dns-slides/slides/start.html">DNS协议与应用</a></p>
<p>比较无语的是js控制翻页这块。我的笔记本键盘只能捕获中间正常键位信号,上下左右、home、end、pgon、pgdn这些统统不行。无奈只好改用输入法一样的逗号,句号.翻页了……</p>
【Logstash系列】使用原版Ruby1.8运行logstash的客户端程序
2012-05-31T00:00:00+08:00
logstash
ruby
http://chenlinux.com/2012/05/31/use-ruby-1.8-for-logstash-agent
<p>在一般情况下,我们实验logstash都是直接用官网上下载的jar包,然后java运行即可。但如果在大规模场景下,这样其实并不是运维的最佳实践:</p>
<ol>
<li>并不是所有设备都默认或者很方便的可以安装java;</li>
<li>默认使用的JRuby执行效率比MRI的版本低一些,为了大规模运维管理,一般部署puppet的时候附加yum/apt获取的是Ruby1.8.7。</li>
</ol>
<p>花了一点时间了解了一下代码结构,发现这点其实是可以做到的。从github上clone代码,在其中的example、bin和lib目录中都看到大量对应官网文档的input/filter/output的东东。</p>
<p>根据我对logstash的了解,仅保留input的file、syslog和remote_luby,filter里的grok,output里的elasticsearch和rabbitmq。然后看lib/logstash/*的具体模块,只有三个模块提到了必须使用java后台。</p>
<p>于是第一步,修改Gemile,只留下必备的模块。<br />
第二步,通过bundle管理工具加载安装。<br />
第三步,通过命令行方式指定配置变量和参数。<br />
第四步,把所属包打包发送到其他设备测试。</p>
<p>现在保留的Gemfile如下:<br />
```ruby<br />
source :rubygems</p>
<p>gem “cabin”, “0.4.4” # for logging. apache 2 license<br />
gem “bunny” # for amqp support, MIT-style license<br />
gem “uuidtools” # for naming amqp queues, License ???</p>
<p>gem “filewatch”, “0.3.3” # for file tailing, BSD License<br />
gem “jls-grok”, “0.10.6” # for grok filter, BSD License<br />
gem “json<br />
gem “mail”</p>
<p>gem “minitest” # License: Ruby</p>
<p>gem “statsd-ruby”, “0.3.0” # outputs/statsd, # License: As-Is</p>
<p>group :test do<br />
gem “mocha”<br />
gem “shoulda”<br />
end<br />
```<br />
然后客户端的运行,这里有点小问题,默认要求必须大于Ruby1.9.2的版本才行。但是通读一遍,发现其实只是用到了Ruby1.9.2里一个全局变量RUBY_ENGINE来判断自己是不是JRuby,这个对等判断很容易修改成为RUBY_DESCRIPTION变量的正则匹配判断。之后就OK了。</p>
<p>具体替换代码如下:<br />
<code class="highlighter-rouge">ruby
# if RUBY_ENGINE == 'JRuby'
if RUBY_DESCRIPTION =~ m/^Ruby/
</code></p>
<p>最后把挑好的lib/*.rb和bin/logstash、etc/logstash打包发送到其他设备。运行也没问题。写上不同的server和agent.conf启动起来一看,果然就传输过去了。<br />
目前就到这步,随后随时更新</p>
用perl和lua在nginx中验证url
2012-05-25T00:00:00+08:00
nginx
lua
perl
http://chenlinux.com/2012/05/25/nginx-auth-by-perl-and-lua
<p>和三年前的博客一样,还是时间加密钥加路径的加密方式。不过这次改用nginx,这样不用重新缓存后面的squid文件了。<br />
先用ngx_lua做:<br />
```nginx<br />
set $expire “600”;<br />
set $salt “mysalt”;<br />
location ~* .mp3$ {<br />
#local m = ngx.re.match(ngx.var.uri,”^/([0-9]{4})/([0-9]{2})/([0-9]{2})/([0-9]{2})/([0-9]{2})/([0-9a-z]{32})(/.*)”)<br />
#用ngx.re.match就不能%d,用string.match就不能{2},郁闷<br />
#而且ngx.re.match所有的捕获都在m数组里,这点类似perl的m//返回。<br />
rewrite_by_lua ‘<br />
local date = {}<br />
local md5str<br />
local path<br />
date.year,date.month,date.day,date.hour,date.min,md5str,path = string.match(ngx.var.uri,”^/(%d+)/(%d+)/(%d+)/(%d+)/(%d+)/(%w+)(/%S+)”)</p>
<div class="highlighter-rouge"><pre class="highlight"><code> if date.year == nil then
ngx.exit(404)
end
local time1 = tonumber(os.time(date))
local time2 = tonumber(ngx.time())
if md5str == ngx.md5(ngx.var.salt..date.year..date.month..date.day..date.hour..date.min..path) then
if time2 - time1 < tonumber(ngx.var.expire) then
ngx.req.set_uri(path)
else
ngx.exit(405)
end
else
ngx.exit(403)
end
';
proxy_pass http://backend;
proxy_set_header Host $host;
} ```
</code></pre>
</div>
<p>然后用nginx_perl做:<br />
```nginx<br />
perl_require POSIX.pm;<br />
perl_require Digest/MD5.pm;<br />
perl_set $realurl ‘<br />
sub {<br />
my $secret = “mysalt”;<br />
my $expire = 600;<br />
my $r = shift;<br />
if ( $r->uri =~ m#^/(\d{4})/(\d{2})/(\d{2})/(\d{2})/(\d{2})/(\w{32})(/\S+.mp3)#oi ) {<br />
my ($year, $mon, $mday, $hour, $min, $md5, $path) = ($1, $2, $3, $4, $5, $6, $7);<br />
my $str = Digest::MD5::md5_hex($secret . $year . $mon . $mday . $hour . $min . $path);<br />
my $reqtime = POSIX::mktime(00, $min, $hour, $mday, $mon - 1, $year - 1900);<br />
my $now = time;<br />
if ( $str eq $md5 and $now - $reqtime < $expire ) {<br />
return $path;<br />
} else {<br />
return “error”;<br />
};<br />
} else {<br />
return “error”;<br />
};<br />
}’;</p>
<p>server { <br />
location ~* .mp3$ {<br />
if ($realurl = “error”) { return 403; }<br />
proxy_pass http://music_store_local$realurl;<br />
proxy_set_header Host $host;<br />
}<br />
}<br />
```</p>
<p>然后用http_load测试单url响应情况,基本效率一样。在压力比较大(lo上跑大概200MB/s,再大就可能出错了)的情况下,第一字节响应时间大概比直接请求squid多一个数量级(从0.4ms到4ms) <br />
这个情况下,squid的cpu%在130%,nginx_perl的worker是25%,nginx_lua的是19%。</p>
【puppet系列】网页展示puppet的客户端报告
2012-05-22T00:00:00+08:00
devops
puppet
dancer
perl
http://chenlinux.com/2012/05/22/puppet-reports-analysis-by-web
<p>上篇说到怎样使用ENC脚本控制puppet的客户端配置,这篇说如何监控和展示客户端运行状态报告。</p>
<p>目前还是使用puppet默认的store方式,也就是报告都存在/var/lib/puppet/reports/$host/$dates.yaml里。所以分析只要针对这个目录下的文件即可,主要使用File::Stat和File::Find两个模块搞定。注意这两个模块在Perl5.16里是默认内核模块了~~</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/perl</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">autodie</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">File::</span><span class="nv">Stat</span> <span class="sx">qw/:stat/</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">YAML::</span><span class="nv">Syck</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">AnyEvent::Filesys::</span><span class="nv">Notify</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">EV</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$interval</span> <span class="o">=</span> <span class="s">"60"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$watch_dir</span> <span class="o">=</span> <span class="s">"/var/lib/puppet/reports"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$periodic</span> <span class="o">=</span> <span class="nn">EV::</span><span class="nv">periodic</span> <span class="mi">0</span><span class="p">,</span> <span class="nv">$interval</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$dirs</span> <span class="o">=</span> <span class="nv">watchdir</span><span class="p">(</span><span class="nv">$interval</span><span class="p">);</span>
<span class="nv">process</span><span class="p">(</span> <span class="nv">$_</span><span class="p">,</span> <span class="s">'No reports return.'</span> <span class="p">)</span> <span class="k">for</span> <span class="nv">@</span><span class="p">{</span><span class="nv">$dirs</span><span class="p">};</span>
<span class="p">};</span>
<span class="k">my</span> <span class="nv">$notifier</span> <span class="o">=</span> <span class="nn">AnyEvent::Filesys::</span><span class="nv">Notify</span><span class="o">-></span><span class="k">new</span><span class="p">(</span>
<span class="nv">dirs</span> <span class="o">=></span> <span class="p">[</span> <span class="nv">$watch_dir</span><span class="p">,</span> <span class="p">],</span>
<span class="nv">interval</span> <span class="o">=></span> <span class="mf">0.5</span><span class="p">,</span>
<span class="nv">filter</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span> <span class="nb">shift</span> <span class="o">=~</span> <span class="sr">/\.yaml$/</span> <span class="p">},</span>
<span class="nv">cb</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">@events</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span> <span class="nv">@events</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$_</span><span class="o">-></span><span class="nv">type</span> <span class="o">=~</span> <span class="sr">m/^(created|modified)$/</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$file</span> <span class="o">=</span> <span class="nv">$_</span><span class="o">-></span><span class="nv">path</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$logs</span> <span class="o">=</span> <span class="nv">LoadFile</span><span class="p">(</span><span class="nv">$file</span><span class="p">)</span><span class="o">-></span><span class="p">{</span><span class="s">'logs'</span><span class="p">};</span>
<span class="k">for</span> <span class="p">(</span> <span class="nv">@</span><span class="p">{</span><span class="nv">$logs</span><span class="p">}</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="s">'level'</span><span class="p">}</span> <span class="ow">eq</span> <span class="s">'err'</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">process</span><span class="p">(</span> <span class="nv">$file</span><span class="p">,</span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="s">'message'</span><span class="p">}</span> <span class="p">);</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="p">},</span>
<span class="p">);</span>
<span class="nn">EV::</span><span class="nv">loop</span><span class="p">();</span>
<span class="k">sub </span><span class="nf">process</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$path</span><span class="p">,</span> <span class="nv">$message</span> <span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$path</span> <span class="o">=~</span> <span class="sr">m/\/([^\/]+\.opi\.com)/</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">print</span> <span class="nv">$1</span><span class="p">,</span><span class="s">" has err: "</span><span class="p">,</span><span class="nv">$message</span><span class="p">,</span><span class="s">"\n"</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="k">sub </span><span class="nf">watchdir</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$interval</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$dirs</span> <span class="o">=</span> <span class="nb">grep</span> <span class="p">{</span> <span class="nb">time</span> <span class="o">-</span> <span class="nb">stat</span><span class="p">(</span><span class="nv">$_</span><span class="p">)</span><span class="o">-></span><span class="p">[</span><span class="mi">9</span><span class="p">]</span> <span class="o">></span> <span class="nv">$interval</span> <span class="p">}</span> <span class="nb">glob</span><span class="p">(</span><span class="s">"${watch_dir}/*"</span><span class="p">);</span>
<span class="k">return</span> <span class="nv">$dirs</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<p>上面这个脚本,通过libev的timer和io分别完成对diretory的mtime的遍历和对file的inotify的监听。process作为公共处理函数,可以随意改造成sms/email/msn等等方式。 <br />
定时器没有用AnyEvent的封装,因为没看到AE有periodic,只有timer。而在io运行的时候,timer是中断的。如果不停有文件inotify发生,timer就没法进行了……periodic的方式与timer不同,是绝对定时而不是相对定时——虽然我个人的浅薄理解觉得应该也被io阻塞,但试验结果是OK的。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/perl</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">autodie</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Dancer</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Template</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">YAML::</span><span class="nv">Syck</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">File::</span><span class="nv">Find</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">File::</span><span class="nv">Stat</span> <span class="sx">qw/:stat/</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">POSIX</span> <span class="sx">qw/strftime/</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Data::Section::</span><span class="nv">Simple</span> <span class="sx">qw/get_data_section/</span><span class="p">;</span>
<span class="nv">set</span> <span class="nv">port</span> <span class="o">=></span> <span class="s">"8080"</span><span class="p">;</span>
<span class="nv">set</span> <span class="nv">daemon</span> <span class="o">=></span> <span class="mi">1</span><span class="p">;</span>
<span class="nv">set</span> <span class="nv">logger</span> <span class="o">=></span> <span class="s">'console'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$tt</span> <span class="o">=</span> <span class="nv">Template</span><span class="o">-></span><span class="k">new</span><span class="p">(</span>
<span class="c1"># DEBUG => 'all',</span>
<span class="p">);</span>
<span class="k">my</span> <span class="nv">$ds_check</span> <span class="o">=</span> <span class="nv">get_data_section</span><span class="p">(</span><span class="s">'check.tt'</span><span class="p">);</span>
<span class="nv">get</span> <span class="s">'/'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$html</span> <span class="o">=</span> <span class="s">'<form action="/ppcheck">'</span><span class="p">;</span>
<span class="nv">$html</span> <span class="o">.=</span> <span class="s">'Write an interval minutes for reports timestamp check: '</span><span class="p">;</span>
<span class="nv">$html</span> <span class="o">.=</span> <span class="s">'<input type="text" name="interval"/> <input type="submit" value="submit" />'</span><span class="p">;</span>
<span class="nv">$html</span> <span class="o">.=</span> <span class="s">'</form>'</span><span class="p">;</span>
<span class="k">return</span> <span class="nv">$html</span><span class="p">;</span>
<span class="p">};</span>
<span class="nv">get</span> <span class="s">'/ppcheck'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">$interval</span> <span class="o">=</span> <span class="nv">params</span><span class="o">-></span><span class="p">{</span><span class="s">'interval'</span><span class="p">}</span> <span class="o">*</span> <span class="mi">60</span> <span class="o">||</span> <span class="mi">300</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$watch_dir</span> <span class="o">=</span> <span class="s">"/var/lib/puppet/reports"</span><span class="p">;</span>
<span class="k">return</span> <span class="s">"Too large number"</span> <span class="k">if</span> <span class="nv">$interval</span> <span class="o">></span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">24</span><span class="p">;</span> <span class="c1"># one day</span>
<span class="k">my</span> <span class="nv">$context</span> <span class="o">=</span> <span class="p">{};</span>
<span class="nv">$context</span><span class="o">-></span><span class="p">{</span><span class="s">'dirs'</span><span class="p">}</span> <span class="o">=</span> <span class="nv">watch_timeout</span><span class="p">(</span> <span class="nv">$interval</span><span class="p">,</span> <span class="nv">$watch_dir</span> <span class="p">);</span>
<span class="nv">$context</span><span class="o">-></span><span class="p">{</span><span class="s">'logs'</span><span class="p">}</span> <span class="o">=</span> <span class="nv">watch_errlogs</span><span class="p">(</span> <span class="nv">$interval</span><span class="p">,</span> <span class="nv">$watch_dir</span> <span class="p">);</span>
<span class="k">my</span> <span class="nv">$output</span><span class="p">;</span>
<span class="nv">$tt</span><span class="o">-></span><span class="nv">process</span><span class="p">(</span><span class="o">\</span><span class="nv">$ds_check</span><span class="p">,</span> <span class="nv">$context</span><span class="p">,</span> <span class="o">\</span><span class="nv">$output</span><span class="p">);</span>
<span class="c1"># $tt->process(\*DATA, $context, \$output);</span>
<span class="c1"># seek *DATA, 1234, 0;</span>
<span class="k">return</span> <span class="nv">$output</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">sub </span><span class="nf">watch_timeout</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span> <span class="nv">$interval</span><span class="p">,</span> <span class="nv">$watch_dir</span> <span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@dirs</span> <span class="o">=</span> <span class="nb">grep</span> <span class="p">{</span> <span class="nb">time</span> <span class="o">-</span> <span class="nb">stat</span><span class="p">(</span><span class="nv">$_</span><span class="p">)</span><span class="o">-></span><span class="p">[</span><span class="mi">9</span><span class="p">]</span> <span class="o">></span> <span class="nv">$interval</span> <span class="p">}</span> <span class="nb">glob</span><span class="p">(</span><span class="s">"${watch_dir}/*"</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">@ret</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span> <span class="nv">@dirs</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$dirtime</span> <span class="o">=</span> <span class="nv">strftime</span><span class="p">(</span><span class="s">"%F %T"</span><span class="p">,</span> <span class="nb">localtime</span><span class="p">(</span><span class="nb">stat</span><span class="p">(</span><span class="nv">$_</span><span class="p">)</span><span class="o">-></span><span class="p">[</span><span class="mi">9</span><span class="p">]));</span>
<span class="k">my</span> <span class="nv">$dirname</span> <span class="o">=</span> <span class="nv">$1</span> <span class="k">if</span> <span class="nv">$_</span> <span class="o">=~</span> <span class="nv">s</span><span class="c1">#([^/]+\.opi\.com)#$1#;</span>
<span class="nb">push</span> <span class="nv">@ret</span><span class="p">,</span> <span class="p">{</span><span class="nv">name</span> <span class="o">=></span> <span class="nv">$dirname</span><span class="p">,</span> <span class="nb">time</span> <span class="o">=></span> <span class="nv">$dirtime</span><span class="p">,</span> <span class="p">};</span>
<span class="p">};</span>
<span class="k">return</span> <span class="o">\</span><span class="nv">@ret</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">sub </span><span class="nf">watch_errlogs</span> <span class="p">{</span>
<span class="k">my</span><span class="p">(</span> <span class="nv">$interval</span><span class="p">,</span> <span class="nv">$watch_dir</span> <span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span><span class="p">(</span> <span class="nv">$wanted</span><span class="p">,</span> <span class="nv">$list_reporter</span> <span class="p">)</span> <span class="o">=</span> <span class="nv">find_file_by_mtime</span><span class="p">(</span><span class="nv">$interval</span><span class="p">);</span>
<span class="nn">File::Find::</span><span class="nv">find</span><span class="p">(</span> <span class="nv">$wanted</span><span class="p">,</span> <span class="nv">$watch_dir</span> <span class="p">);</span>
<span class="k">my</span> <span class="nv">@ret</span> <span class="o">=</span> <span class="nv">$list_reporter</span><span class="o">-></span><span class="p">();</span>
<span class="k">return</span> <span class="o">\</span><span class="nv">@ret</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">sub </span><span class="nf">find_file_by_mtime</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$interval</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@found</span> <span class="o">=</span> <span class="p">();</span>
<span class="k">my</span> <span class="nv">$finder</span> <span class="o">=</span> <span class="k">sub </span><span class="p">{</span>
<span class="k">if</span> <span class="p">(</span> <span class="o">-</span><span class="nv">f</span> <span class="nv">$</span><span class="nn">File::Find::</span><span class="nv">name</span> <span class="o">&&</span> <span class="nb">time</span> <span class="o">-</span> <span class="nb">stat</span><span class="p">(</span><span class="nv">$</span><span class="nn">File::Find::</span><span class="nv">name</span><span class="p">)</span><span class="o">-></span><span class="p">[</span><span class="mi">9</span><span class="p">]</span> <span class="o"><</span> <span class="nv">$interval</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$yaml</span> <span class="o">=</span> <span class="nn">YAML::Syck::</span><span class="nv">LoadFile</span><span class="p">(</span><span class="nv">$</span><span class="nn">File::Find::</span><span class="nv">name</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">@logs</span> <span class="o">=</span> <span class="nb">grep</span> <span class="p">{</span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="s">'level'</span><span class="p">}</span> <span class="ow">eq</span> <span class="s">'err'</span> <span class="p">}</span> <span class="nv">@</span><span class="p">{</span><span class="nv">$yaml</span><span class="o">-></span><span class="p">{</span><span class="s">'logs'</span><span class="p">}};</span>
<span class="k">for</span> <span class="p">(</span> <span class="nv">@logs</span> <span class="p">)</span> <span class="p">{</span>
<span class="nb">push</span> <span class="nv">@found</span><span class="p">,</span> <span class="p">{</span> <span class="nv">host</span> <span class="o">=></span> <span class="nv">$yaml</span><span class="o">-></span><span class="p">{</span><span class="s">'host'</span><span class="p">},</span> <span class="nv">message</span> <span class="o">=></span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="s">'message'</span><span class="p">},</span> <span class="p">};</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="k">my</span> <span class="nv">$reporter</span> <span class="o">=</span> <span class="k">sub </span><span class="p">{</span> <span class="nv">@found</span> <span class="p">};</span>
<span class="k">return</span><span class="p">(</span> <span class="nv">$finder</span><span class="p">,</span> <span class="nv">$reporter</span> <span class="p">);</span>
<span class="p">};</span>
<span class="nv">dance</span><span class="p">;</span>
<span class="bp">__DATA__</span>
<span class="nv">@@</span> <span class="nv">check</span><span class="o">.</span><span class="nv">tt</span>
<span class="o"><</span><span class="nv">div</span> <span class="nv">id</span><span class="o">=</span><span class="s">"timeoutdirs"</span><span class="o">></span>
<span class="nv">List</span> <span class="nv">of</span> <span class="nv">nodes</span> <span class="nv">whose</span> <span class="nv">report</span> <span class="nv">is</span> <span class="nv">timeout:</span> <span class="o"><</span><span class="nv">br</span> <span class="o">/></span>
<span class="sr"><ul></span>
<span class="p">[</span><span class="nv">%</span> <span class="nv">FOREACH</span> <span class="nv">dir</span> <span class="nv">IN</span> <span class="nv">dirs</span> <span class="nv">%</span><span class="err">]</span>
<span class="err"><</span><span class="nv">li</span> <span class="nv">style</span><span class="o">=</span><span class="s">"width:200px;float:left"</span><span class="o">></span><span class="p">[</span><span class="nv">%</span> <span class="nv">dir</span><span class="o">.</span><span class="nv">name</span> <span class="nv">%</span><span class="err">]</</span><span class="nv">li</span><span class="o">><</span><span class="nv">li</span> <span class="nv">style</span><span class="o">=</span><span class="s">"width:200px;margin:0;float:left"</span><span class="o">></span><span class="p">[</span><span class="nv">%</span> <span class="nv">dir</span><span class="o">.</span><span class="nb">time</span> <span class="nv">%</span><span class="err">]</</span><span class="nv">li</span><span class="o">></span>
<span class="p">[</span><span class="nv">%</span> <span class="nv">END</span> <span class="nv">%</span><span class="err">]</span>
<span class="err"></</span><span class="nv">ul</span><span class="o">></span>
<span class="sr"></div></span>
<span class="o"><</span><span class="nv">br</span> <span class="sr">/><hr /</span><span class="o">><</span><span class="nv">br</span> <span class="o">/></span>
<span class="o"><</span><span class="nv">div</span> <span class="nv">id</span><span class="o">=</span><span class="s">"runerrlogs"</span><span class="o">></span>
<span class="nv">Error</span> <span class="nv">messages</span> <span class="nv">of</span> <span class="nv">running</span> <span class="nv">nodes:</span> <span class="o"><</span><span class="nv">br</span> <span class="o">/></span>
<span class="sr"><ul></span>
<span class="p">[</span><span class="nv">%</span> <span class="nv">FOREACH</span> <span class="nb">log</span> <span class="nv">IN</span> <span class="nv">logs</span> <span class="nv">%</span><span class="err">]</span>
<span class="err"><</span><span class="nv">li</span> <span class="nv">style</span><span class="o">=</span><span class="s">"width:200px;float:left"</span><span class="o">></span><span class="p">[</span><span class="nv">%</span> <span class="nv">log</span><span class="o">.</span><span class="nv">host</span> <span class="nv">%</span><span class="err">]</</span><span class="nv">li</span><span class="o">></span>
<span class="o"><</span><span class="nv">li</span> <span class="nv">style</span><span class="o">=</span><span class="s">"width:400px;margin:0;float:left"</span><span class="o">></span><span class="p">[</span><span class="nv">%</span> <span class="nv">log</span><span class="o">.</span><span class="nv">message</span> <span class="nv">%</span><span class="err">]</</span><span class="nv">li</span><span class="o">></span>
<span class="p">[</span><span class="nv">%</span> <span class="nv">END</span> <span class="nv">%</span><span class="err">]</span>
<span class="err"></</span><span class="nv">ul</span><span class="o">></span>
<span class="sr"></div></span>
</code></pre>
</div>
<p>这个脚本实现的功能其实和上面那个类似。不过报警改成web页面,event触发改成web请求触发。 <br />
这里两个新难点:</p>
<p>其一是没有inotify后如何根据web请求参数查找范围内新建的报表。File::Find模块只有一个函数find(\&wanted,@dirs)。其中&wanted是不能传参进去的。 <br />
在CPAN上看到一个叫做File::Find::Closures的模块,提供了一系列可以给File::Find使用的&wanted函数,包括一个示例。于是稍微改造一下,就写成了find_file_by_mtime()函数。</p>
<p>其二是因为偷懒没有用dancer建立完整项目,使用了perl virtual file来提供template。所以不能直接使用Dancer的template ‘ttname’, {var=>$var};定义了。 <br />
Template::Toolkit提供的process()可以操作的template来源很多,可以是字符串/文件/句柄。所以process(*DATA)也是生效的。但是问题出现了:这样启动后,第一次访问正常;第二次访问返回空! <br />
打开Template的DEBUG看到,第二次访问的时候,从*DATA里读不到数据了。也就是说,必须重新seek回去——而且试验证明seek的起始点是shebang行。。。。 <br />
所以只能先从__DATA__里读出数据,然后以字符串形式传递给process()了。</p>
<p>这里用到了Data::Section模块。从CloudForecast项目里学来的。CloudForecast中的web页面,使用的Text::Xslate模板技术读取__DATA__。其中包括有index/server/servers/service等页面。也就是说,在一个__DATA__里实现了多个virtual file。用的就是Data::Section::Simple模块。以@@为标签分割即可。</p>
【puppet系列】puppet使用ENC定义节点
2012-05-18T00:00:00+08:00
devops
perl
puppet
http://chenlinux.com/2012/05/18/puppet-external-nodes-classifier
<p>今天研究puppet dashboard。主要有ENC和reports两个功能。其中ENC功能相当扯淡,因为你在web上点击添加的class/node/group,是没有任何依赖性检查(比如node命名是否符合fqdn,class是否存在)的,随便咋填绝无报错和拒绝!而且也没有提供类似report的导入工具,一旦启用就要完全重新手工输入所有配置……所以无论是从导入角度还是管理角度,自己实现一个靠谱点的ENC都是有必要的。</p>
<p>关于puppet的ENC配置,参见<a href="http://docs.puppetlabs.com/guides/external_nodes.html">ENC的文档</a></p>
<p>主要就是修改puppet.conf里两个配置:</p>
<ul>
<li><code class="highlighter-rouge">node_terminus</code>,由plain修改成exec;</li>
<li><code class="highlighter-rouge">external_nodes</code>,由none修改为ENC脚本的路径。</li>
</ul>
<p>类似如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[master]
node_terminus = exec
external_nodes = /etc/puppet/webui/external_nodes
</code></pre>
</div>
<p>脚本输入输出的说明:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Its only argument is the name of the node to be classified, and it returns a YAML document describing the node.
</code></pre>
</div>
<p>注意到此为止配置修改不算结束!文档中提到,puppet是支持同时开启ENC和site.pp配置的。puppet会自动merge两个配置。但是debug运行时可以看到,这个merge是按照node级别进行的。也就是说:</p>
<ol>
<li>puppet master收到一个node的请求,如果 <code class="highlighter-rouge">node_terminus</code> 配置为exec,输出node的fqdn给 <code class="highlighter-rouge">external_nodes</code>;</li>
<li>收到 <code class="highlighter-rouge">external_nodes</code> 的返回,为一个yaml体或者空;</li>
<li>加载site.pp,这是按照文本顺序进行的。如果都是import “module”,那么最后就进入module的parameter和template处理。</li>
<li>如果已经存在,检查是全局变量还是类,全局变量的话报错,类的话覆盖为site.pp中最后定义的类。 <br />
<strong>2013年4月10日更新:感谢<a href="http://weibo.com/liucy1983">@liu.cy</a>指出这里变量和类的区别</strong></li>
</ol>
<p>所以为了方便起见,请删除掉site.pp中的<code class="highlighter-rouge">import "node/*.pp"</code> 这行配置。我在这里就被郁闷了很久。</p>
<p>然后是我这里的想法是尽量不更改pp的语法,只是提供一个把group里的ip到fqdn的转化然后查找cluster配置组合成yaml。 <br />
也就是说有一个group的配置目录,其配置文件为”groupname.pp”,内容如下:<br />
<code class="highlighter-rouge">ruby
group "groupname" {
$string = "test"
$arrays = [123, abc]
include module1, module2
}
</code><br />
还有一个iplist的配置目录,其配置文件为“groupname.iplist”,内容如下:<br />
<code class="highlighter-rouge">squid
1.2.3.4
2.3.4.5
</code><br />
那么以后服务器组有啥变更,只需要修改一下iplist就好了,不用重启puppet进程。</p>
<p>所以有两个脚本,一个是 <code class="highlighter-rouge">external_node</code> 脚本:<br />
```perl<br />
#!/bin/env perl<br />
use warnings;<br />
use strict;<br />
use autodie;<br />
use DBI;<br />
use YAML::Syck;</p>
<p>my $base_dir = “/etc/puppet/webui”;</p>
<p>my $node = $ARGV[0];<br />
my $group = sqlite_select($node);<br />
my $hash = pp2hashref($group);<br />
print Dump($hash);<br />
# ENC要求退出值必须为0 <br />
exit 0;</p>
<p>sub pp2hashref {<br />
my $group = shift;<br />
my $data = {};<br />
open my $fh, ‘<’, “${base_dir}/group/${group}.pp”;<br />
my $i;<br />
while(<$fh>) {<br />
if ( /^#|^\s<em>$|}$/ ) {<br />
next;<br />
} elsif ( /^group\s+”(\w+)”/ ) {<br />
die “group name do not match,check please!” unless $1 =~ m/$group/i;<br />
} elsif ( /^\s</em>?$(\w+)\s<em>=\s</em>”(.+)”$/ ) {<br />
$data->{“parameters”}->{“$1”} = $2;<br />
} elsif ( /^\s<em>?include\s+(.+)$/ ) {<br />
@{$data->{“classes”}} = split(/,\s</em>/, $1);<br />
} elsif ( /^\s<em>$(\w+)\s</em>=\s<em>[([^]]+)]?/ ) {<br />
$i = $1;<br />
grep { push @{$data->{“parameters”}->{“$i”}}, $_ } split(/,\s</em>/, $2);<br />
} elsif ( /^\s<em>([^]]+)]?/ ) {<br />
grep { push @{$data->{“parameters”}->{“$i”}}, $_ } split(/,\s</em>/, $1);<br />
} else {<br />
next;<br />
};<br />
};<br />
$data->{“parameters”}->{“clustername”} = $group;<br />
# ENC要求输出的yaml中必须提供environment参数 <br />
$data->{“environment”} = “production”;<br />
return $data;<br />
}</p>
<p>sub sqlite_select {<br />
my $node = shift;<br />
my $dbh = DBI->connect(“dbi:SQLite:dbname=${base_dir}/node_info.db”,””,””,{RaiseError=>1,AutoCommit=>0});<br />
my $sth = $dbh->prepare(“select node_group from node_info where node_fqdn = ?”);<br />
$sth->execute(“$node”);<br />
my $ret = $sth->fetchrow_hashref->{“node_group”};<br />
$dbh->disconnect();<br />
return $ret;<br />
};<br />
<code class="highlighter-rouge">
一个是维护sqlite的脚本:
</code>perl<br />
#!/bin/env perl<br />
use warnings;<br />
use strict;<br />
use autodie;<br />
use DBI;<br />
use Net::Nslookup;</p>
<p>my $base_dir = “/etc/puppet/webui”;</p>
<p>sqlite_update();</p>
<p>sub ip_conv {<br />
my $ip = shift;<br />
my $name = nslookup(host => “$ip”, type => “PTR”);<br />
# ENC的输入参数为全小写格式,所以sqlite中也必须存储小写格式的主机名 <br />
return lc($name);<br />
};</p>
<p>sub sqlite_rebuild {<br />
# 配置系统变动不大,且puppet本身还有一层也是用sqlite的node配置缓存层,<br />
# 所以这里不用复杂的select判断再update或者insert,直接重建sqlite <br />
unlink “${base_dir}/node_info.db”;<br />
my $dbh = DBI->connect(“dbi:SQLite:dbname=${base_dir}/node_info.db”,””,””,{RaiseError=>1,AutoCommit=>0});<br />
my $sql = ‘create table node_info (node_fqdn, node_group)’;<br />
$dbh->do($sql);<br />
# sqlite支持简单事务,所以要即时提交 <br />
$dbh->commit();<br />
my $sth = $dbh->prepare(‘replace into node_info values(?,?)’);<br />
my @groups = grep { s/^${base_dir}\/iplist\/(\w+?).list$/$1/ } glob(“${base_dir}/iplist/*”);<br />
print $<em>,”\n” for @groups;<br />
foreach ( @groups ) {<br />
my $group = $</em>;<br />
open my $fh, ‘<’, “${base_dir}/iplist/${group}.list”;<br />
while (<$fh>) {<br />
my $fqdn = ip_conv($_);<br />
$sth->execute(“$fqdn”, “$group”);<br />
die $DBI::errstr if $dbh->err();<br />
};<br />
};<br />
$dbh->commit();<br />
$dbh->disconnect();<br />
};<br />
<code class="highlighter-rouge">
以上是ENC的配置。继续分析puppet dashboard,除了ENC外,另一个功能就是reports,相比ENC来说,reports功能还算稍微靠谱一点,用http方式替换puppet自身的store方式,并存数据在mysql里。目前使用dashboard的人主要也就是在用这个功能。
但是我个人认为,一般情况下运维不可能专门开一个页面看着puppet,也不太会有必要按照时间段查看状态报表汇总图这个东东,真正要紧的,是及时接到运行错误的报警以便上机处理。所以这里最后是一个监控reports的脚本,目前还没看到http方式的reports数据格式,所以暂时继续使用store的方式,然后采用Linux文件系统的inotify方式报警。
</code>perl<br />
#!env perl<br />
use strict;<br />
use warnings;<br />
use YAML::Syck;<br />
use AnyEvent::Filesys::Notify;<br />
use EV;</p>
<h1 id="aelinuxinotify2bsd">使用AE的这个扩展而不直接用Linux::Inotify2模块,方便万一之后迁移到BSD主机</h1>
<p># 而且异步回调的方式性能更好,在压力较大时不会阻塞丢失事件 <br />
my $notifier = AnyEvent::Filesys::Notify->new(<br />
dirs => [qw(/var/lib/puppet/reports)],<br />
interval => 0.5,<br />
# 在puppetd请求的时候,会在目录下先生成临时文件,完成后再mv成正式的,所以要过滤<br />
filter => sub { shift =~ /.yaml$/ },<br />
cb => sub {<br />
for ( @_ ) {<br />
# 一般这里会是两个type,创建的created和修改的modified,因为文件名只精确到分钟,如果两次运行在一分钟内,文件名就一样<br />
if ( $<em>->type ) {<br />
my $file = $</em>->path;<br />
my $logs = LoadFile($file)->{‘logs’};<br />
for ( @{$logs} ) {<br />
if ( $<em>->{‘level’} eq ‘err’ ) {<br />
process($file, $</em>->{‘message’});<br />
};<br />
};<br />
};<br />
};<br />
},<br />
);</p>
<p>EV::loop();</p>
<p>sub process {<br />
my ( $path, $message ) = @_;<br />
if ( $path =~ m/\/([^\/]+)\/\d{12}.yaml$/ ) {<br />
# 这里用nagios还是email方式处理都可以,代码略<br />
print $1,” has err: “,$message,”\n”;<br />
};<br />
};</p>
<p>```</p>
【puppet系列】puppet安装/Facter插件和puppet模板编写
2012-05-10T00:00:00+08:00
devops
puppet
ruby
http://chenlinux.com/2012/05/10/quick-start-for-puppet-facter-erb
<p>使用puppet管理集群配置是个很靠谱的做法。跟其他同类产品相比,第一他的DSL语法很丰富够灵活,第二围绕他的生态圈活跃,资料比较多。</p>
<h1 id="puppet">puppet安装</h1>
<p>由于关注热度比较高,所以各种简便安装办法都出来了。大家可以各自选择,走yum、apt或者src都行。我这里演示一下用rubygems的办法。优点是不用像yum那样找epel源,而且版本够新,缺点是没有yum自动出来的配置目录和管理脚本。<br />
<code class="highlighter-rouge">bash
#!/usr/bin/env bash
# Set Curlrc for https
echo 'insecure' >> ~/.curlrc
# Install git
curl -L get-git.rvm.io | sudo bash
# Install RVM
curl -L get.rvm.io | sudo bash -s stable
# Install Last Ruby
source "/usr/local/rvm/scripts/rvm"
rvm install 1.9.3
# Use GEM Mirror in Taobao
gem sources -r http://rubygems.org/
gem sources -a http://rubygems.taobao.org/
# Install puppet
gem install puppet --no-ri --no-rdoc
groupadd puppet
useradd -g puppet -s /bin/false -M puppet
# Get default puppet config
mkdir /etc/puppet
puppet --genconfig > /etc/puppet/puppet.conf
</code><br />
需要说明一点,puppet对集群的识别高度依赖fdqn,所以必须保证主机名和IP的一一对应。我的环境里因为本身kerberos认证也是fdqn依赖的,所以反而省事不少,其他环境中,估计折腾dns或者hosts也是个大步骤。</p>
<h1 id="facter">facter简介</h1>
<p>在安装puppet的时候,会顺带安装好facter工具,这个工具是用来探测本机各类变量,提供给puppetd使用的。</p>
<p>因为facter也是ruby写的。所以我们可以自己根据其规则书写补充工具获取更多的变量,方便之后定制模块和类资源。</p>
<h1 id="puppetca">puppetca认证</h1>
<p>安装完成后需要建立认证关系。puppet没有像其他通用系统一样借用sshkey的认证,而是自己维护了一套,所以有这么个单独的章节:</p>
<p>首先是客户端上的申请,没有单独的命令,就跟一次正常请求一样即可: <br />
<code class="highlighter-rouge">bash
puppetd --test --server master.pp.domain.com
</code></p>
<p>注意:因为puppet2.7和ruby1.9.2以上版本在ssl上的冲突,所以新版本ruby的client需要多几步处理: <br />
<code class="highlighter-rouge">bash
scp master.puppet.domain.com:/etc/puppet/ssl/certs/ca.pem /etc/puppet/ssl/certs/
hash=`openssl x509 -hash -noout -in /etc/puppet/ssl/certs/ca.pem`
ln -s /etc/puppet/ssl/certs/ca.pem /etc/pki/tls/certs/${hash}.0
</code></p>
<p>这样才能正常申请cert。</p>
<p>然后在主控端上审批。首先可以puppetca –list查看有多少请求过来的client是未认证的。然后运行如下命令通过: <br />
<code class="highlighter-rouge">bash
puppetca -s -a
</code><br />
(吐槽一下,内网上已经有这么多认证了,还搞一套pp的,还搞出问题来了,有够无聊的,强烈希望pp提供一个开关)</p>
<h1 id="sitepp">site.pp简介</h1>
<p>/etc/puppet/manifests/site.pp是puppetmaster的总入口。在这里主要完成几个工作:加载模块(import “module”);加载节点配置(node {})。</p>
<p>模块的名字,就是直接来自于/etc/puppet/modules/dir_name的命名;而node里include包含的class类名,则是modules/dir_name/manifests/init.pp中写的名字。</p>
<p>一般来说,这两个可以保持一致方便管理。但是规则毕竟如此,碰到不一致的还是要懂(我就是碰上不一致的样例才研究的)。</p>
<p>node配置太多的话,可以另外写在别的目录和配置文件里。在site.pp中只要<code class="highlighter-rouge">include 'nodes/*.pp'</code>这样就可以了。</p>
<p>node配置主要三个部分:</p>
<ol>
<li>node的fdqn的正则匹配;</li>
<li>node的变量赋值;</li>
<li>node的特定类;</li>
</ol>
<p>比如下面这个例子:<br />
<code class="highlighter-rouge">ruby
node "cache[0-9]\.domain\.com" {
$var = "strings"
$array = ["one", "two"]
include facter squid
}
</code></p>
<h1 id="modules">modules简介</h1>
<p>模块中一定有manifests/init.pp作为入口,然后根据其中的定义,另外有templates和files作为辅助目录。</p>
<p>init.pp中可以使用require ‘other.pp’指令来加载模块的其他配置。</p>
<p>init.pp中可以使用puppet的各种资源,包括file、service、exec、package、cron、user等等。</p>
<p>各种资源下有各种属性和方法可用,这里就不展开讲了,一来我也没掌握多少,二来真写全了够出书了。主要写关于file的几个,因为linux的本质就是All is file嘛~</p>
<ol>
<li>content:直接在puppet配置里写file的内容,一般只会在测试的时候这么直接用。更多的是用template(‘class/class.template.erb’)的方式调用模板文件。</li>
<li>source:直接下载puppetmaster上的原始文件,具体url写法是puppet:///module/file。</li>
<li>notify:在客户端搞定file的写入后准备触发的下一个操作,一般用来重启服务(可以用service方法,也可以用exec方法)等等。</li>
<li>ensure:file的状态。比如absent是如果已存在的就删除掉;present是如果不存在就新建;directory是目录;latest是最新的包版本等等。</li>
<li>path:file在客户端的真是存放路径。如果没有定义,就是用name配置。</li>
<li>mode:file在客户端的权限,也就是644,755之类的。</li>
</ol>
<h1 id="template">template简介</h1>
<p>puppet用的erb模板引擎,也是RoR中用的模板引擎。看起来和Perl5中的Template::Toolkit相当的一样。尤其是<%%>里使用=和-的表现。</p>
<p>我们可以在puppet里用erb完成比puppet语法复杂的多的功能,比如根据node的变量进行运算、循环、判断等等。</p>
<h1 id="squid">squid示例</h1>
<p>最后举例今天完成的一个简单的squid的配置。</p>
<p>首先是/etc/puppet/manifests/site.pp: <br />
```ruby<br />
import “squid”<br />
node “cache[0-9].domain.com” {</p>
<p>$cache_peers = [ ‘1.2.3.4’, ‘1.2.3.5’]<br />
$http_port = “8080”<br />
$fs_type = “aufs”<br />
# 单独区分coss,是因为squid源码被修改过后,关于COSS的配置跟磁盘都是特制的,不能像原版那样计算了 <br />
# $fs_type = “coss”<br />
# $coss_options = “/data/coss/stripe0 32 backstore=/data/coss/stripe4,32 max-size=1024768 block-size=1024”</p>
<p>include facter squid<br />
}<br />
```</p>
<p>然后是/etc/puppet/modules/facter/manifests/init.pp:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="n">facter</span> <span class="p">{</span>
<span class="n">file</span> <span class="p">{</span> <span class="s2">"df.rb"</span><span class="p">:</span>
<span class="n">path</span> <span class="o">=></span> <span class="s2">"/usr/lib/ruby/gems/1.8/gems/facter-1.6.8/lib/facter/df.rb"</span><span class="p">,</span>
<span class="k">ensure</span> <span class="o">=></span> <span class="n">file</span><span class="p">,</span>
<span class="n">mode</span> <span class="o">=></span> <span class="mi">644</span><span class="p">,</span>
<span class="n">source</span> <span class="o">=></span> <span class="s2">"puppet:///modules/facter/df.rb"</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>关于给facter写插件,网站的资料说的path都是直接在#{rubysitedir}/facter目录下。但我这里实际情况却不是(好吧,暴露了我的实验机并没有按照上面说的用rvm安装ruby而是yum的)。 <br />
<strong>2013年1月31日更正:</strong><br />
更优的做法应该是放在<code class="highlighter-rouge">#{puppetdir}/modules/yourmodule/lib/facter/</code>下,然后作为pluginsync发下去。<br />
<strong>更正以上</strong></p>
<p>然后是对应的/etc/puppet/modules/facter/files/df.rb:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">data_dir_list</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">df</span> <span class="o">=</span> <span class="no">Facter</span><span class="o">::</span><span class="no">Util</span><span class="o">::</span><span class="no">Resolution</span><span class="p">.</span><span class="nf">exec</span><span class="p">(</span><span class="s2">"/bin/df -m 2>/dev/null"</span><span class="p">)</span>
<span class="n">df</span><span class="p">.</span><span class="nf">each_line</span> <span class="k">do</span> <span class="o">|</span><span class="n">l</span><span class="o">|</span>
<span class="k">if</span> <span class="n">l</span> <span class="o">=~</span> <span class="sr">/^\/dev\/\w+\s+(\d+).+\/(data\d*)$/</span>
<span class="n">data_dir_list</span><span class="p">[</span><span class="vg">$2</span><span class="p">]</span> <span class="o">=</span> <span class="vg">$1</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="no">Facter</span><span class="p">.</span><span class="nf">add</span><span class="p">(</span><span class="s2">"DataDirCount"</span><span class="p">)</span> <span class="k">do</span>
<span class="n">setcode</span> <span class="k">do</span>
<span class="k">if</span> <span class="n">data_dir_list</span><span class="p">.</span><span class="nf">length</span> <span class="o">!=</span> <span class="mi">0</span>
<span class="n">data_dir_list</span><span class="p">.</span><span class="nf">length</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">data_dir_list</span><span class="p">.</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">k</span><span class="p">,</span><span class="n">v</span><span class="o">|</span>
<span class="no">Facter</span><span class="p">.</span><span class="nf">add</span><span class="p">(</span><span class="s2">"DirSize_</span><span class="si">#{</span><span class="n">k</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span> <span class="k">do</span>
<span class="n">setcode</span> <span class="k">do</span>
<span class="n">v</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p>实现很简单,实质就是执行df -m,获取挂载点为/data、/data1…的目录数以及各目录的总大小,然后把结果添加到facter里。之所以要加这么个插件,是因为之后squid的缓存目录,需要根据目录数量和大小自动计算,而标准的facter里没有这方面的信息,无法传递相关变量。</p>
<p>下面正式进入squid模块部分。看/etc/puppet/modules/squid/manifests/init.pp:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="n">squid</span> <span class="p">{</span>
<span class="n">service</span> <span class="p">{</span> <span class="s2">"squid"</span><span class="p">:</span>
<span class="k">ensure</span> <span class="o">=></span> <span class="n">running</span><span class="p">,</span>
<span class="n">subscribe</span> <span class="o">=></span> <span class="no">File</span><span class="p">[</span><span class="s2">"squid.conf"</span><span class="p">],</span>
<span class="p">}</span>
<span class="n">file</span> <span class="p">{</span> <span class="s2">"squid.conf"</span><span class="p">:</span>
<span class="n">path</span> <span class="o">=></span> <span class="s2">"/tmp/squid.conf"</span><span class="p">,</span>
<span class="n">notify</span> <span class="o">=></span> <span class="no">Service</span><span class="p">[</span><span class="s2">"squid"</span><span class="p">],</span>
<span class="n">content</span> <span class="o">=></span> <span class="n">template</span><span class="p">(</span><span class="s2">"squid/squid.conf.erb"</span><span class="p">),</span>
<span class="k">ensure</span> <span class="o">=></span> <span class="n">present</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>这里只写了service和file两个,实际上还应该有package保证client上确实有squid软件,有file保证/etc/init.d/squid脚本存在等等。注意其中file里的notify和service里的subscribe正好是对应的意思。</p>
<p>最后是/etc/puppet/squid/template/squid.conf.erb:<br />
```squid<br />
<% if (fs_type == ‘aufs’) -%><br />
cache_dir aufs /data/fcache <%= Integer(dirsize_data.to_i<em>0.8) %> 16 256<br />
<% if (datadircount.to_i > 1) -%><br />
<% 1.upto(datadircount.to_i - 1).each do |i| -%><br />
<% size = eval “dirsize_data#{i}” -%><br />
cache_dir aufs /data<%= i %>/fcache <%= Integer(size.to_i</em>0.8) %> 16 256<br />
<% end -%><br />
<% end -%><br />
<% else %><br />
cache_dir coss <%= coss_options %><br />
<% end %><br />
cache_mem <%= Integer(memorysize.to_i * 0.45 * 1024) %> MB<br />
visible_hostname <%= fqdn %><br />
http_port <%= http_port %> vhost</p>
<p><% cache_peers.each do |peer| -%><br />
cache_peer <%= peer %> parent 80 0 no-query originserver round-robin<br />
<% end %><br />
```<br />
这里只贴跟模板变量相关的部分。初学ruby,被to_i方法搞得很是郁闷,还好像eval方法之类的很像很眼熟~~ <br />
模板里cache_peers等,是在node配置里定义的;memorysize等,是facter获取的。</p>
通过lua统计nginx内部变量数据
2012-05-08T00:00:00+08:00
nginx
lua
http://chenlinux.com/2012/05/08/collect-log-variables-by-ngx-lua
<p>统计nginx的请求数据,一般有几个办法,一个是logrotate,通过access.log计算,这个很详细,但是实时性差一些;一个是Tengine提供的pipe,这个实时性更好,但是管道如果出现堵塞,麻烦就多了~这两种办法,归根结底都是把日志记录在本地(pipe方式如果要长期保留依然要记磁盘)然后由脚本完成计算。今天这里说另一种方法:在nginx内部,随着每次请求完成一些基础的数据统计,然后输出到存储里供长期调用。 <br />
代码如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">server</span> <span class="p">{</span>
<span class="kn">listen</span> <span class="mi">80</span><span class="p">;</span>
<span class="kn">server_name</span> <span class="s">photo.domain.com</span><span class="p">;</span>
<span class="kn">location</span> <span class="n">/</span> <span class="p">{</span>
<span class="kn">set</span> <span class="nv">$str</span> <span class="nv">$uri</span><span class="p">;</span>
<span class="kn">content_by_lua</span> <span class="s">'</span>
<span class="s">local</span> <span class="s">url</span> <span class="p">=</span> <span class="s">ngx.var.uri</span>
<span class="s">local</span> <span class="s">res</span> <span class="p">=</span> <span class="s">ngx.location.capture("/proxy",</span> <span class="p">{</span><span class="kn">vars</span> <span class="p">=</span> <span class="p">{</span> <span class="kn">str</span> <span class="p">=</span> <span class="s">url</span> <span class="err">}}</span><span class="s">)</span>
<span class="s">ngx.print(res.body)</span>
<span class="s">ngx.shared.log_dict:set("url",</span> <span class="s">url)</span>
<span class="s">local</span> <span class="s">upstream_stat</span> <span class="p">=</span> <span class="s">ngx.var.status</span>
<span class="s">local</span> <span class="s">upstream_time</span> <span class="p">=</span> <span class="s">tonumber(ngx.var.upstream_response_time)</span>
<span class="s">local</span> <span class="s">redis</span> <span class="p">=</span> <span class="s">require</span> <span class="s">"resty.redis"</span>
<span class="s">local</span> <span class="s">red</span> <span class="p">=</span> <span class="s">redis:new()</span>
<span class="s">local</span> <span class="s">ok,</span> <span class="s">err</span> <span class="p">=</span> <span class="s">red:connect("127.0.0.1",</span> <span class="mi">6379</span><span class="s">)</span>
<span class="s">if</span> <span class="s">upstream_stat</span> <span class="p">~</span><span class="sr">=</span> <span class="s">"200"</span> <span class="s">then</span>
<span class="s">red:sadd("url",url)</span>
<span class="s">red:incr(url)</span>
<span class="s">red:incr(url..":time",</span> <span class="s">upstream_time)</span>
<span class="s">end</span>
<span class="s">'</span><span class="p">;</span>
<span class="p">}</span>
<span class="kn">location</span> <span class="n">/dict_status</span> <span class="p">{</span>
<span class="kn">content_by_lua</span> <span class="s">'</span>
<span class="s">local</span> <span class="s">url</span> <span class="p">=</span> <span class="s">ngx.shared.log_dict:get("url")</span>
<span class="s">ngx.say(url)</span>
<span class="s">'</span><span class="p">;</span>
<span class="p">}</span>
<span class="kn">location</span> <span class="n">/redis_status</span> <span class="p">{</span>
<span class="kn">content_by_lua</span> <span class="s">'</span>
<span class="s">local</span> <span class="s">redis</span> <span class="p">=</span> <span class="s">require</span> <span class="s">"resty.redis"</span>
<span class="s">local</span> <span class="s">red</span> <span class="p">=</span> <span class="s">redis:new()</span>
<span class="s">local</span> <span class="s">ok,err</span> <span class="p">=</span> <span class="s">red:connect("127.0.0.1",</span> <span class="mi">6379</span><span class="s">)</span>
<span class="s">local</span> <span class="s">urlist,err</span> <span class="p">=</span> <span class="s">red:sort("url","limit","0","1","desc","by","*")</span>
<span class="s">if</span> <span class="s">not</span> <span class="s">urlist</span> <span class="s">then</span>
<span class="s">ngx.say(err)</span>
<span class="s">return</span>
<span class="s">end</span>
<span class="s">for</span> <span class="s">i</span> <span class="p">=</span> <span class="mi">1</span><span class="s">,</span> <span class="c1">#urlist do
</span> <span class="s">local</span> <span class="s">avg</span> <span class="p">=</span> <span class="s">red:get(urlist[i])</span>
<span class="s">local</span> <span class="s">sum</span> <span class="p">=</span> <span class="s">red:get(urlist[i]..":time")</span>
<span class="s">ngx.say(urlist[i],"</span><span class="err">\</span><span class="s">tavg_time:",avg/sum,</span> <span class="s">"</span><span class="err">\</span><span class="s">tsum:",sum)</span>
<span class="s">end</span>
<span class="s">'</span><span class="p">;</span>
<span class="p">}</span>
<span class="kn">location</span> <span class="n">/proxy</span> <span class="p">{</span>
<span class="kn">proxy_pass</span> <span class="s">http://backend_fmn_xnimg_cn</span><span class="nv">$str</span><span class="p">;</span>
<span class="kn">proxy_set_header</span> <span class="s">Host</span> <span class="s">'fmn.rrimg.com'</span><span class="p">;</span>
<span class="kn">include</span> <span class="s">conf.d/proxy.conf</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>ngx_lua里的指令有set/rewrite/header_filter/log/content/access_by_lua等,它们各自处于nginx处理流程中的某一步,所以有些日志变量可能不一定都能读取到。还有header_filter和log两个不能调用subrequest和output的API(也就是只能使用上例代码中的ngx.shared.DICT方式,但只支持简单的key-value),content不能和proxy_pass在一起等等……</p>
<p>不过content里可以调用ngx.location.capture()来subrequest其他location,比如这里利用/proxy来完成原来的proxy_pass的功能。 <br />
因为subrequest后$uri有变化,所以pass必须写对真正的url的全路径。这就靠之前的set $str来传递变量了。</p>
<p>最终运行结果:</p>
<div class="highlighter-rouge"><pre class="highlight"><code># curl http://fmn.rrimg.cn/redis_status
/test avg_time:0.73 sum:12
</code></pre>
</div>
Linux系统调优读书笔记
2012-04-30T00:00:00+08:00
linux
tunning
monitor
ruby
hadoop
http://chenlinux.com/2012/04/30/reading-notes-about-linux-system
<p>今天在图书馆看书,摘抄一些有意思的细节。</p>
<h1 id="linux">Linux服务器性能调整</h1>
<p>Linux内存布局NUMA: <br />
非一致性读取 每个节点; <br />
每个节点下有 多个管理区(ZONE)内存块; <br />
内存块包括: <br />
ZONE_DMA 0~16MB <br />
ZONE_NORMAL 16~896MB<br />
ZONE_HIGHMEM 896MB~结束 <br />
32位处理器下,用户空间3GB,内核空间1GB; <br />
内核空间除去ZONE_DMA和ZONE_NORMAL后只剩下128MB用来vmalloc/kmap等操作; <br />
kmap操作用来虚拟化页表数据位,可以在32位处理器下支持64GB内存。 <br />
NUMA结构下8节点的互连结构,只是3个相互最近的互连; <br />
不同节点之间的定时器很难一致,通常选一个做唯一定时。</p>
<p>多处理SMP: <br />
松耦合系统:每个处理器都有自己的总线、内存和IO系统; <br />
紧耦合系统:只运行一个OS; <br />
对称系统均分任务; <br />
非对称系统有一个主控处理器; <br />
cache是否共享?内存、总线和IO子系统肯定是共享的,cache会带来一致性问题; <br />
锁竞争导致的开销,所以N个处理器不能达到N倍效率提升; <br />
affinity即绑定进程到CPU,让进程跟cache更近。 <br />
linux里进程是高权的线程(一般说法是反过来说线程是小型的进程)</p>
<p>集群cluster: <br />
高性能:分布式任务并行处理,100+节点,通称为计算机; <br />
高可用:故障问题,最多16节点,通常2~4节点,通称为企业服务器。</p>
<p>系统跟踪前提: <br />
容量足够;开销较小;场景可重现;尽量无其他进程干扰除非是场景本身需要。</p>
<p>strace命令场景: <br />
判断IO阻塞,内存分配及其频率等。</p>
<p>OProfile:<br />
opcontrol命令:设置的count要足够大,否则中断本身次数会影响结果。</p>
<p>内核调度器: <br />
优先级0~MAX_PRIO(140); <br />
0~100为实时任务; <br />
101~140为分时任务,即nice命令调整的-20~19。</p>
<p>I/O调度器: <br />
deadline: <br />
read_expire; <br />
seek_cost=(x+stream_unit-1)/stream_unit,默认stream_unit为4字节; <br />
write_starved:读优先N次后才写;</p>
<p>文件系统: <br />
hdparm:MaxMultSect参数,默认16,当前都支持32位输出了。</p>
<p>网络: <br />
tcp_window_scaling、tcp_sack、tcp_fsack等。</p>
<p>进程间通信: <br />
ipcs -u查看状态; <br />
ipcs -l查看限制。</p>
<p>数据库: <br />
OLTP在线事务处理的业务类型类似小文件,一般文件块大小在2KB左右; <br />
DSS决策支持系统的业务类型类似大文件,一般文件块大小在8KB以上。</p>
<h1 id="section">网站性能监测与优化</h1>
<p>Netflix的Jiffy是客户端收集的开源项目; <br />
sqmphoniq即是服务器端分析,也要客户端js的支持; <br />
Episodes同上。</p>
<h1 id="jruby">JRuby语言实战技术</h1>
<p>类与超类: <br />
所有类的超类都是Object; <br />
所有类都是类Class的对象。</p>
<h1 id="hadoop">Hadoop实战</h1>
<p>chukwa监控系统: <br />
在HDFS和Map/Reduce的基础上,即意味着日志非tail方式,而是有collector先行合并成大文件再存储到HDFS; <br />
致力于2000+节点,1TB+日志量的集群日志分析,即意味着非实时性; <br />
数据结果目前还是存MySQL里再web展示,可能会转移到HBase上。 <br />
流程架构类似如下: <br />
N个agents(每台server一个) –HTTP–> M个collectors(每百个agent一个) –sink–> HDFS –map/reduce–> MySQL <–JSP/JS–> Web(HICC) <br />
整个流程跟logstash很类似,但是logstash没有固定hadoop平台,所以不用sink这步,实时性更好;而agent的input代码段就类似于执行类似map的工作,output段就是reduce结果了。</p>
51CTO博客自动发布脚本
2012-04-21T00:00:00+08:00
perl
http://chenlinux.com/2012/04/21/backup-blog-to-51cto
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/bin/env perl</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">File::</span><span class="nv">Util</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">YAML::</span><span class="nv">Syck</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Perl6::</span><span class="nv">Say</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">XMLRPC::</span><span class="nv">Lite</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Data::</span><span class="nv">Dumper</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$f</span> <span class="o">=</span> <span class="nn">File::</span><span class="nv">Util</span><span class="o">-></span><span class="k">new</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@blogs</span> <span class="o">=</span> <span class="nb">grep</span> <span class="p">{</span><span class="sr">/\.markdown$/</span><span class="p">}</span> <span class="nv">$f</span><span class="o">-></span><span class="nv">list_dir</span><span class="p">(</span><span class="s">'../_posts'</span><span class="p">,</span> <span class="s">'--recurse'</span><span class="p">);</span>
<span class="k">foreach</span> <span class="p">(</span><span class="nv">@blogs</span><span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$yaml</span> <span class="o">=</span> <span class="nv">LoadFile</span><span class="p">(</span><span class="nv">$_</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$title</span> <span class="o">=</span> <span class="nv">$yaml</span><span class="o">-></span><span class="p">{</span><span class="s">'title'</span><span class="p">};</span>
<span class="k">my</span> <span class="nv">$text</span> <span class="o">=</span> <span class="nv">$f</span><span class="o">-></span><span class="nv">load_file</span><span class="p">(</span><span class="s">"$_"</span><span class="p">);</span>
<span class="nv">upload</span><span class="p">(</span><span class="nv">$title</span><span class="p">,</span> <span class="nv">$text</span><span class="p">);</span>
<span class="p">};</span>
<span class="k">sub </span><span class="nf">upload</span> <span class="p">{</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$title</span><span class="p">,</span> <span class="nv">$text</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$username</span> <span class="o">=</span> <span class="s">'username'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$password</span> <span class="o">=</span> <span class="s">'password'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$blogid</span> <span class="o">=</span> <span class="s">'123456'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$proxyurl</span> <span class="o">=</span> <span class="s">'http://blogname.blog.51cto.com/xmlrpc.php'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$res</span> <span class="o">=</span> <span class="nn">XMLRPC::</span><span class="nv">Lite</span><span class="o">-></span><span class="nv">proxy</span><span class="p">(</span><span class="nv">$proxyurl</span><span class="p">)</span><span class="o">-></span><span class="nv">call</span><span class="p">(</span><span class="s">'metaWeblog.newPost'</span><span class="p">,</span> <span class="nv">$blogid</span><span class="p">,</span> <span class="nv">$username</span><span class="p">,</span> <span class="nv">$password</span><span class="p">,</span> <span class="p">{</span> <span class="nv">title</span> <span class="o">=></span> <span class="s">"$title"</span><span class="p">,</span> <span class="nv">description</span> <span class="o">=></span> <span class="s">"$text"</span><span class="p">,</span> <span class="nv">categories</span> <span class="o">=></span> <span class="p">[</span><span class="s">'【创作类型:原创】'</span><span class="p">,</span><span class="s">'IT管理'</span><span class="p">,</span> <span class="p">]},</span> <span class="mi">1</span><span class="p">)</span><span class="o">-></span><span class="nv">result</span><span class="p">;</span>
<span class="nv">say</span> <span class="s">"newPost id -- "</span> <span class="o">.</span> <span class="nv">$res</span> <span class="k">if</span> <span class="nv">$res</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<p>目前还有几个问题:</p>
<ol>
<li>虽然写了创作类型是原创,但是发布后还是转载;</li>
<li>categories的数组生成的xml格式是<array><data><value><base64>,但通过wireshark抓包windows live writer看到能被api接收的格式应该是<array><data><value><string>。在XMLRPC::Lite的代码中,有as_base64/as_string等好几个方法,但是找到从哪里定义使用,目前简单的采用注释掉了XMLRPC::Lite::Serializer::new()方法里的base64键值对,但是理论上应该不用修改源码的;</string></value></data></array></base64></value></data></array></li>
<li>最后一个最关键的问题,不管我在服务器上是用utf8还是gb2312,上传后都是乱码。估计第一个问题其实也是由此产生的。</li>
</ol>
获取造价百强公司的真实位置
2012-04-18T00:00:00+08:00
perl
http://chenlinux.com/2012/04/18/get-location-of-some-companys
<p>很久没更新,没用技术,今天稍微geek一下下。给老婆搜索她行业百强公司的具体地点,看看如果换单位的话是否方便出行~代码如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/perl</span>
<span class="k">use</span> <span class="nn">Data::</span><span class="nv">Dumper</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">LWP::</span><span class="nv">UserAgent</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">URI</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Web::</span><span class="nv">Scraper</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">JSON::</span><span class="nv">XS</span><span class="p">;</span>
<span class="c1"># 处理中文需要指定输入输出都用utf8格式,否则会有wide character in print的warning提示 </span>
<span class="k">use</span> <span class="nv">utf8</span><span class="p">;</span>
<span class="nb">binmode</span><span class="p">(</span><span class="bp">STDIN</span><span class="p">,</span> <span class="s">':encoding(utf8)'</span><span class="p">);</span>
<span class="nb">binmode</span><span class="p">(</span><span class="bp">STDOUT</span><span class="p">,</span> <span class="s">':encoding(utf8)'</span><span class="p">);</span>
<span class="nb">binmode</span><span class="p">(</span><span class="bp">STDERR</span><span class="p">,</span> <span class="s">':encoding(utf8)'</span><span class="p">);</span>
<span class="c1"># 百度地图搜索的查询结果返回的是json数据,需要转换成perl的哈希格式 </span>
<span class="k">sub </span><span class="nf">decode_map_json</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$map_json</span> <span class="o">=</span> <span class="nv">decode_json</span> <span class="nb">shift</span><span class="p">;</span>
<span class="c1"># 如果是距离搜索,那么$map_json->{"content"}->[0]->{"lines"}->[0]->[2]是距离;$map_json->{"taxi"}->{"detail"}->[0]->{"totalPrice"}是打的费用。</span>
<span class="k">return</span> <span class="nv">$map_json</span><span class="o">-></span><span class="p">{</span><span class="s">"content"</span><span class="p">}</span><span class="o">-></span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">-></span><span class="p">{</span><span class="s">"addr"</span><span class="p">};</span>
<span class="p">};</span>
<span class="c1"># 用LWP发起地图查询请求 </span>
<span class="k">sub </span><span class="nf">get_map_json</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$company</span> <span class="o">=</span> <span class="nv">$_</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$ua</span> <span class="o">=</span> <span class="nn">LWP::</span><span class="nv">UserAgent</span><span class="o">-></span><span class="k">new</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$res</span> <span class="o">=</span> <span class="nv">$ua</span><span class="o">-></span><span class="nv">get</span><span class="p">(</span><span class="s">'http://map.baidu.com/?qt=s&wd='</span><span class="o">.</span><span class="nv">$company</span><span class="p">);</span>
<span class="k">return</span> <span class="nv">$res</span><span class="o">-></span><span class="nv">decoded_content</span> <span class="k">if</span> <span class="nv">$res</span><span class="o">-></span><span class="nv">is_success</span><span class="p">;</span>
<span class="p">};</span>
<span class="c1"># 采用Web::Scraper获取网页里的XPath内容 </span>
<span class="k">sub </span><span class="nf">get_company</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$ori_uri</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="c1"># 可以在firefox里直接查看元素的XPath,在google chrome里则需要安装Xpath Helper工具。</span>
<span class="c1"># 安装完成后,使用Ctrl+Shift+X快捷键呼出顶端Xpath调试框,然后按住Shift键,用鼠标左键点击网页元素,上边框里就出现元素的Xpath和content了。</span>
<span class="c1"># 注意复制过来的Xpath里的@会被perl理解成是数组的标示,所以要加上逃逸符\才行。</span>
<span class="k">my</span> <span class="nv">$tweets</span> <span class="o">=</span> <span class="nv">scraper</span> <span class="p">{</span>
<span class="nv">process</span> <span class="s">"/html/body/div[\@class='index-main layout']/div[\@class='index-main layout']/div[\@class='index-content bcolor']/div[\@class='cont']/div[\@id='fontzoom']/span[\@id='BodyLabel']/div/table"</span><span class="p">,</span> <span class="s">"list"</span> <span class="o">=></span> <span class="nv">scraper</span> <span class="p">{</span>
<span class="c1"># 第一个scraper里不能获取tbody,原因未知。所以分成两步,先获取一个到table的scraper,再获取tbody里面的TEXT。</span>
<span class="nv">process</span> <span class="s">"tbody tr td:nth-child(2)"</span><span class="p">,</span> <span class="s">'cont[]'</span> <span class="o">=></span> <span class="s">'TEXT'</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="k">my</span> <span class="nv">$res</span> <span class="o">=</span> <span class="nv">$tweets</span><span class="o">-></span><span class="nv">scrape</span><span class="p">(</span><span class="nv">URI</span><span class="o">-></span><span class="k">new</span><span class="p">(</span><span class="nv">$ori_uri</span><span class="p">));</span>
<span class="k">return</span> <span class="nv">$res</span><span class="o">-></span><span class="p">{</span><span class="s">'list'</span><span class="p">}</span><span class="o">-></span><span class="p">{</span><span class="s">'cont'</span><span class="p">};</span>
<span class="p">};</span>
<span class="k">my</span> <span class="nv">$company</span> <span class="o">=</span> <span class="nv">get_company</span><span class="p">(</span><span class="s">'http://www.ceca.org.cn/show.aspx?id=2006'</span><span class="p">);</span>
<span class="k">foreach</span><span class="p">(</span><span class="nv">@$company</span><span class="p">){</span>
<span class="k">print</span> <span class="nv">$_</span><span class="p">,</span><span class="s">"\t"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$json</span> <span class="o">=</span> <span class="nv">get_map_json</span><span class="p">(</span><span class="nv">$_</span><span class="p">);</span>
<span class="k">print</span> <span class="nv">decode_map_json</span><span class="p">(</span><span class="nv">$json</span><span class="p">),</span><span class="s">"\n"</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<p>输出结果如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>单位名称
上海东方投资监理有限公司 中国·上海市江宁路1306弄7号富丽大厦23楼
中冶京诚工程技术有限公司 白广路4号
中铁工程设计咨询集团有限公司
中冶赛迪工程技术股份有限公司 重庆市渝中区双钢路一号
中国电力工程顾问集团西南电力设计院
上海第一测量师事务所有限公司 澳门路519弄1-5号
中竞发(北京)工程造价咨询有限公司 知春路108号豪景大厦A座13层
</code></pre>
</div>
<p>搜狗地图的api只是js的,不方便弄。不然直接获取从当前地点到目的地点的距离和耗时就更好了~~百度地图页面上就没看到有api提供,虽然用着,还是鄙视一下~~</p>
弹性集群监控中的配置自动生效问题研究
2012-04-16T00:00:00+08:00
monitor
nagios
gearman
perl
http://chenlinux.com/2012/04/16/configs-automatically-effective-for-elastic-monitor
<p>最近跟<a href="http://weibo.com/fedoracore">@画圈圈的星星</a> 聊天,说到nagios在大规模集群运用中一个比较严重的瓶颈:修改配置需要重启进程。 <br />
听起来似乎不是什么问题,我个人之前对nagios的追求,也都放在怎么样提供一个及时高效的监控和数据展示上面—-这两个问题在 <code class="highlighter-rouge">mod_gearman</code> 和 <code class="highlighter-rouge">pnp4nagios</code> 的协助下已经很给力了。</p>
<p>但是聊天中提到了一个新的场景,事实上早在两个月前,<a href="http://weibo.com/tjpm">@GNUer</a> 就提到过类似的场景,就是当 nagios 监控的是这样一个弹性集群的时候:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>集群设备数以千记,甚至是上万的规模;
而且设备上运行着复杂的应用,每台设备都有几十上百的监控项需求;
为了提供高可靠性,集群以资源池的方式运行,即设备随时可能更改当前应用角色,在idle/lb/cache/web/app/db/storage等之间切换;
</code></pre>
</div>
<p>以上。</p>
<p>尤其是最后一条,假如这个更改频率快到了每分钟都有变更,那么 nagios 重启进程这点就足以打死它了。实际运行中我们可以知道,当 <code class="highlighter-rouge">nagios reload</code> 的时候,这个命令的执行本身就要花费远大于1分钟的时间。</p>
<p>临时的办法,就是在更改后不主动 reload,而是在 crontab 里定时去做。损失一些监控实时性。一般来说,还不至于真的同一台设备一分钟内连续更改角色并且需要分别监控的。</p>
<p>但是真要做到实时,应该怎么做呢?</p>
<p>首先想到的是 <code class="highlighter-rouge">mod_gearman</code> 的基础上进行改造。我们之前已经知道,<code class="highlighter-rouge">mod_gearman</code> 上是可以分别有 host_check/service_check/check_result 几个 jobs 的。那么,我们可以跳过 config 阶段,自己写 gearman client 发送 job 。这一步很容易。难点是 check_result 被 nagios 回收后,我们自己发的 job,其 host/service 在 nagios 的 service_list 结构里是不存在的……所以还要自己写 gearman worker 来回收 result,具体来说,必须要做的事情包括有:根据 performance_data 来 create 和 update 相应的 rrds;根据 exit status 来启动 notification。这个工作内容一下子达到自己重写一个比较完整的监控系统的地步了,而且你如果通过原版的 cgi 查看,这部分内容还查看不到……</p>
<p>于是我在 github 上询问 <code class="highlighter-rouge">mod_gearman</code> 的作者<a href="https://github.com/sni">Sven Nierlein</a> ,他回答说:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>There is no such feature right now and it would be very hard to implement such thing in nagios or icinga.
It should be easier to implement something like that in shinken, but i guess it still takes 2-3 weeks of development.
</code></pre>
</div>
<p>好吧,比较失望的回答。尝试去瞄一眼 nagios-src,在 base/events.c 里可以看到,nagios 是在读取完全部 config 之后,才进入 loop,并提供 eventbroker 的 api 的。</p>
<p>shinken 是完全重写过的披着 nagios 皮的监控系统,在 <a href="http://shinken.ideascale.com/">shinken 的 suggestion 征集页面</a> 上,我看到也有一位提议:<a href="http://shinken.ideascale.com/a/dtd/Arbiter-configuration-without-reloading-daemon/323455-10373">Arbiter configuration without reloading daemon</a>,不过应者寥寥,看来这种需求真的是极少数人才会碰到的。</p>
<p>既然说到用 gearman,又说到监控,再回头看看去年提到的 cloudforecast。在 <code class="highlighter-rouge">ConfigLoader.pm</code> 中,可以看到一个 <code class="highlighter-rouge">watchdog</code> 方法。具体代码如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">my</span> <span class="nv">$watcher</span> <span class="o">=</span> <span class="nn">Filesys::Notify::</span><span class="nv">Simple</span><span class="o">-></span><span class="k">new</span><span class="p">(</span><span class="o">\</span><span class="nv">@path</span><span class="p">);</span>
<span class="k">while</span> <span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$watcher</span><span class="o">-></span><span class="nb">wait</span><span class="p">(</span> <span class="k">sub </span><span class="p">{</span>
<span class="k">my</span> <span class="nv">@path</span> <span class="o">=</span> <span class="nb">grep</span> <span class="p">{</span> <span class="nv">$_</span> <span class="o">!~</span> <span class="sr">m![/\\][\._]|\.bak$|~$!</span> <span class="p">}</span> <span class="nb">map</span> <span class="p">{</span> <span class="nv">$_</span><span class="o">-></span><span class="p">{</span><span class="nv">path</span><span class="p">}</span> <span class="p">}</span> <span class="nv">@_</span><span class="p">;</span>
<span class="k">return</span> <span class="k">if</span> <span class="o">!</span> <span class="nv">@path</span><span class="p">;</span>
<span class="nn">CloudForecast::</span><span class="nv">Log</span><span class="o">-></span><span class="nb">warn</span><span class="p">(</span> <span class="s">"File updates: "</span> <span class="o">.</span> <span class="nb">join</span><span class="p">(</span><span class="s">","</span><span class="p">,</span> <span class="nv">@path</span><span class="p">)</span> <span class="p">);</span>
<span class="nb">sleep</span> <span class="mi">1</span><span class="p">;</span>
<span class="nb">kill</span> <span class="s">'TERM'</span><span class="p">,</span> <span class="nv">$parent_pid</span><span class="p">;</span>
<span class="nb">exit</span><span class="p">;</span>
<span class="p">}</span> <span class="p">);</span>
<span class="p">}</span>
</code></pre>
</div>
<p>可以看到,其实现方法是通过另起进程,通过 inotify 监听文件修改的方式,”实时”的重启主进程。实质上与 nagios 并无区别,都是要重新加载内存中保存的整个监控项配置列表。虽然没有大压力运用,但是可以猜测在预设环境中,重启耗时也会是瓶颈。</p>
<p>另外一个监控系统 zabbix,与上面两个系统都不同,他的监控配置,不是通过文件方式存在监控服务器上,而是通过 web 操作保存在数据库里。整个 host/item/template 等等都是鼠标点击即可。</p>
<p>zabbix 我的使用经验不多,只在三年前用它的预设步骤的方式监控过网页性能。印象中在 create graph 后需要等待一定时间后才能反映出结果。但不确定这个时间是监控项排队消耗的,还是监控进程重启消耗掉的。</p>
<p>和<a href="http://weibo.com/frankymryao">@超大杯摩卡星冰乐</a> 询问了一下,只能猜测或许是通过循环 scan table 的方式”实时”的添加”新”监控项到监控进程的队列里。或许也得跟分析 nagios 一样看看代码才知道是否能在本文预设的弹性环境下适用了。</p>
PostgreSQL中国用户会DBA2000培训计划北京第二课笔记
2012-03-18T00:00:00+08:00
database
PostgreSQL
http://chenlinux.com/2012/03/18/postgreSQL-DBA-2000-note2
<h2 id="section">运行维护</h2>
<h3 id="vacuum">vacuum命令</h3>
<p>pgsql是multi-version concurrency control的,update和delete的操作并不会真正的修改原版本的内容,而只是做一个标记,最后需要用vacuum命令回收失效版本的位置。 <br />
vacuum的主要作用:<br />
1. 恢复或重用失效的空间;<br />
2. 更新pgsql规划器的数据统计;<br />
3. 避免事务ID的重复。<br />
事务ID只有32位,差不多40亿左右。建议在达到10亿左右的时候就需要vacuum一次。 <br />
在version8.*之后,默认就是用auto vacuum。注意auto vacuum不是定时启动,而是触发式的。</p>
<p>vacuum命令有两种形式:<br />
1. vacuum,正常情况,不阻塞读写。<br />
2. vacuum full,使用全表排他锁,不可读,产生最小大小的数据文件。不建议在7*24的生产环境使用。</p>
<p>vacuum full命令的操作原理简述:<br />
1. 标记旧数据;<br />
2. 移动数据成连续空间;<br />
3. 截断文件。</p>
<h3 id="reindex">reindex命令</h3>
<p>在version7.4之后,该命令不再需要经常性运行了。 <br />
执行该命令会阻塞写操作。读操作照常。</p>
<h3 id="analyze">analyze命令</h3>
<p>建议规划一个database范围的analyze,然后每天运行看效果。</p>
<h2 id="section-1">存储过程</h2>
<h3 id="plpgsql">pl/pgsql示例:</h3>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">CREATE</span> <span class="k">FUNCTION</span> <span class="n">func_name</span> <span class="p">(</span> <span class="k">option</span> <span class="k">type</span> <span class="p">)</span> <span class="k">RETURNS</span>
<span class="k">type</span> <span class="k">AS</span> <span class="err">$$</span>
<span class="p">...</span>
</code></pre>
</div>
<h3 id="section-2">触发器示例:</h3>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">CREATE</span> <span class="k">FUNCTION</span> <span class="k">trigger_name</span> <span class="p">(</span> <span class="k">option</span> <span class="k">type</span> <span class="p">)</span> <span class="k">RETURNS</span>
<span class="n">tirgger</span> <span class="k">AS</span> <span class="err">$$</span>
<span class="k">DECLARE</span> <span class="p">...</span>
<span class="k">BEGIN</span>
<span class="p">....</span>
<span class="k">RETURN</span> <span class="k">NEW</span><span class="o">/</span><span class="k">NULL</span> <span class="cm">/*NULL就回滚上面的操作*/</span>
<span class="k">END</span>
</code></pre>
</div>
<h3 id="section-3">调试</h3>
<p>图形化安装时带有的pgadmin3里有一项debugger。 <br />
配置:shared_preload_libraries=”$libdir/plugins/plugin_debugger.so” <br />
导入:debugger.sql</p>
<h2 id="section-4">监控</h2>
<ol>
<li>data/pg_log/*.log</li>
</ol>
<p>标示等级一般为:通用等级LOG NOTICE,错误等级FATAL ERROR,提示等级LOG HINT<br />
一般有一个startup.log文件记录启动过程;一些以时间为名字的日志,记录运行过程,每当文件超过10MB,每次重启,以及每过一整天的时候,就会生成一个新文件。</p>
<ol>
<li>pgadmin3</li>
</ol>
<p>通过server status看锁状态,杀进程等。</p>
<ol>
<li>psql命令</li>
</ol>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">select</span> <span class="o">*</span> <span class="k">from</span> <span class="n">pg_stat_activetity</span><span class="p">;</span>
</code></pre>
</div>
<p>配置:log_min_duration_statement,设置慢查询日志的时限,单位为毫秒。</p>
<h2 id="section-5">集群</h2>
<h3 id="section-6">8.*时代</h3>
<p>复制以WAL File为单位,一旦丢失,就可能损失16MB的事务。而且standby不可读。</p>
<h3 id="section-7">9.*时代</h3>
<p>复制以WAL中的record为单位,且standby可以读操作,能设置成读写分离集群。<br />
9.0中只有异步复制;9.1中有同步复制。</p>
<h3 id="section-8">主要方案</h3>
<p>PGPool II等</p>
PostgreSQL中国用户会DBA2000培训计划北京第一课笔记
2012-03-17T00:00:00+08:00
database
PostgreSQL
http://chenlinux.com/2012/03/17/postgreSQL-DBA-2000-note1
<h1 id="postgresql">PostgreSQL及中国用户会简介</h1>
<p>主讲人 李元佳 galy</p>
<h2 id="section">数据库分类</h2>
<p>商业数据库: Oracle, DB2, SQLserver, Sybase…<br />
开源数据库: MySQL, PostgreSQL, Firebird, SQLite, Apache Derby…</p>
<h2 id="postgresql-1">PostgreSQL沿革</h2>
<p>类BSD许可的,面向对象的,关系型数据库管理系统。</p>
<p>MIT –> Ingres –> Postgres –> PostgreSQL ( 同源的还有SQLserver等 )</p>
<p>支持SQL2008标准的大部分功能特性,是各种RDBMS的SQL方言中最贴近标准的。</p>
<h1 id="postgresql-2">PostgreSQL简介</h1>
<p>主讲人 萧少聪 Scott.Siu</p>
<h2 id="section-1">用户与进程</h2>
<p><img src="/images/uploads/pgsql-process.png" alt="postgreSQL中用户与进程的联系" /></p>
<p>注意在上图中,不管是workmem还是sharebuffer,每个page都是8KB大小。</p>
<h2 id="section-2">复制流程</h2>
<p>stream replica的流程如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>client --> postgres --> WAL (not file)--> slave --> (return OK) --> master --> commit
</code></pre>
</div>
<p>在master上的流程细节如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>client --> write-ahead log(WAL) buffer --> commit --> (async/fsync~~160%) --> WAL Files (16MB * 132个)
^
|--> share buffer --> bgwriter --> db files
^ |
|-- check point <-- ## 安装
</code></pre>
</div>
<p>linux: 注意使用独立的非root用户来安装启动pgsql。在version9.1后,可以跟SElinux结合使用,提高安全性。 <br />
win: 只能在NTFS文件系统上创建表空间。 <br />
窗口统一式安装,可以方便的安装stack builder套件。</p>
<h2 id="section-3">目录</h2>
<p>默认使用窗口安装的情况下,目录结构如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>/opt/PostgreSQL/9.1/
|
|--> bin/
|--> doc/
|--> include/
|--> lib/
|--> share/
|--> install/
|--> data/
|--> base/ 存放table和index的ID号
|--> global/
|--> pg_clog/ 运行日志
|--> pg_xlog/ WAL日志
|--> pg_tblspc/ 表空间ID,实质为到真实数据目录的软连接
|--> postgresql.conf
|--> pg_hba.conf
</code></pre>
</div>
<h2 id="section-4">创建</h2>
<ol>
<li>使用bin/initdb命令;</li>
<li>修改data/pg_hba.conf里的连接地址段和登录权限;</li>
<li>修改data/postgresql.conf里的监听网卡。</li>
</ol>
<h2 id="section-5">启动与停止</h2>
<p>使用bin/pg_ctl命令。其停止命令可指定三种类型:</p>
<ol>
<li>smart模式,即等待全部client连接断开后停止;</li>
<li>fast模式,即直接回滚全部尚未完成的事务后停止;</li>
<li>immediate模式,即立刻中止全部进程。</li>
</ol>
<h2 id="section-6">配置说明</h2>
<ol>
<li>
<p>work_mem: <br />
并不是每个client连接的postgres进程分配一个work mem,而是SQL每一次的排序work使用一个work mem。包括join和order by。如果没有排序,就不用work mem。如果一条sql里同时使用了N次排序,那么就要使用N个work mem。所以理想的使用方法不是提供太大的work mem来排序,而是尽量缩小需要排序的数据大小,设置为4/8MB即可。 <br />
该配置是可以online修改的。命令如下: <br />
SET work_mem = 2048;<br />
SET work_mem = ‘2MB’;<br />
上面两条命令等价。可以看书其计量单位为1KB,且类型为字符串,所以在自定义计量时需要用引号。</p>
</li>
<li>
<p>share_buffers: <br />
理论上为机器物理内存的40%大小。实际测试显示大于8GB后,性能不会有相应的提升,即可认为最大设置到8GB。</p>
</li>
<li>
<p>temp_buffers: <br />
无修改意义</p>
</li>
<li>
<p>max_prepared_transactions: <br />
并发事务数</p>
</li>
<li>
<p>maintenance_work_mem: <br />
vacuum、create index、alter table add foreign key等管理命令使用的work_mem,建议设置1G。因为这些命令经常涉及全表扫描。</p>
</li>
</ol>
<h2 id="postgresql-3">postgreSQL的数据集概念</h2>
<div class="highlighter-rouge"><pre class="highlight"><code> DataBase Cluster
|
|---------|---------|
| | |
user database tablespace
|
schema
</code></pre>
</div>
<p>这里的cluster不是HA cluster,而是数据集。 <br />
一个database里可以有多个schema,一个user可以有多个schema的管理权限,但一个schema只能归属于一个user。 <br />
默认有一个template0为schema的基础,不可修改,在template0基础上有template1,可以修改。实际创建schema时就是复制template1出来。<br />
创建user时,一般都会再创建一个同名的schema,并规定该schema的所属人为该user。这样在pgsql连接到database后,其默认schema即为该同名schema。</p>
<h2 id="section-7">备份与恢复</h2>
<h3 id="section-8">备份</h3>
<p>pg_dump命令,使用-s指定只备份数据结构,-t指定只备份数据内容。</p>
<h3 id="section-9">基于时间点的备份恢复</h3>
<ol>
<li>select pg_start_backup(‘FullBackup’);</li>
<li>tar zcvf full_backup/week1.tgz /opt/PostgreSQL/9.1/data/</li>
<li>
<p>select pg_stop_backup();</p>
</li>
<li>tar zxvf full_backup/week1.tgz -C /</li>
<li>echo ‘restore_command=”cp %f %p”’ > data/recovery.conf</li>
</ol>
加强了解nginx的几个问题
2012-03-10T00:00:00+08:00
nginx
http://chenlinux.com/2012/03/10/go-deep-into-nginx-operation
<p>被问到一些关于nginx或者说nginx运维相关的问题,记录下来几个值得思考的。这里面有些是自己曾经想到过但是浅浅的了解下就不放在心上的,有些是根本没想过这会成为一个”有意思”的问题的……</p>
<h2 id="nginxclientip">1、nginx日志记录得到client的IP原理。</h2>
<p>nginx记录的client的IP分两种,一种是$remote_addr,一种是$http_x_forwarded_for。其中X-Forwarded-For里存放的是proxy加入的client端IP,通过http header传递的。而$remote_addr是TCP上的结果。但是具体如何不知道。今天回来翻nginx的src,先从定义nginx变量的ngx_http_variable.c看到$remote_addr是这样来的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">ngx_http_variable_remote_addr</span><span class="p">(</span><span class="n">ngx_http_request_t</span> <span class="o">*</span><span class="n">r</span><span class="p">,</span>
<span class="n">ngx_http_variable_value_t</span> <span class="o">*</span><span class="n">v</span><span class="p">,</span> <span class="kt">uintptr_t</span> <span class="n">data</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">v</span><span class="o">-></span><span class="n">len</span> <span class="o">=</span> <span class="n">r</span><span class="o">-></span><span class="n">connection</span><span class="o">-></span><span class="n">addr_text</span><span class="p">.</span><span class="n">len</span><span class="p">;</span>
<span class="n">v</span><span class="o">-></span><span class="n">valid</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="n">v</span><span class="o">-></span><span class="n">no_cacheable</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">v</span><span class="o">-></span><span class="n">not_found</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">v</span><span class="o">-></span><span class="n">data</span> <span class="o">=</span> <span class="n">r</span><span class="o">-></span><span class="n">connection</span><span class="o">-></span><span class="n">addr_text</span><span class="p">.</span><span class="n">data</span><span class="p">;</span>
<span class="k">return</span> <span class="n">NGX_OK</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
<p>然后可以在ngx_http.h和ngx_http_request.h里看到</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">typedef</span> <span class="k">struct</span> <span class="n">ngx_http_request_s</span> <span class="n">ngx_http_request_t</span><span class="p">;</span>
<span class="p">...</span>
<span class="k">struct</span> <span class="n">ngx_http_request_s</span> <span class="p">{</span>
<span class="kt">uint32_t</span> <span class="n">signature</span><span class="p">;</span> <span class="cm">/* "HTTP" */</span>
<span class="n">ngx_connection_t</span> <span class="o">*</span><span class="n">connection</span><span class="p">;</span>
<span class="n">ngx_buf_t</span> <span class="o">*</span><span class="n">header_in</span><span class="p">;</span>
<span class="n">ngx_http_headers_in_t</span> <span class="n">headers_in</span><span class="p">;</span>
<span class="n">ngx_http_headers_out_t</span> <span class="n">headers_out</span><span class="p">;</span>
<span class="n">ngx_http_request_body_t</span> <span class="o">*</span><span class="n">request_body</span><span class="p">;</span>
<span class="p">...</span>
<span class="p">}</span>
</code></pre>
</div>
<p>然后在ngx_connection.c里看到</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="cp">#include <ngx_core.h>
</span><span class="p">...</span>
<span class="n">ngx_listening_t</span> <span class="o">*</span>
<span class="n">ngx_create_listening</span><span class="p">(</span><span class="n">ngx_conf_t</span> <span class="o">*</span><span class="n">cf</span><span class="p">,</span> <span class="kt">void</span> <span class="o">*</span><span class="n">sockaddr</span><span class="p">,</span> <span class="n">socklen_t</span> <span class="n">socklen</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">ngx_listening_t</span> <span class="o">*</span><span class="n">ls</span><span class="p">;</span>
<span class="k">struct</span> <span class="n">sockaddr</span> <span class="o">*</span><span class="n">sa</span><span class="p">;</span>
<span class="n">u_char</span> <span class="n">text</span><span class="p">[</span><span class="n">NGX_SOCKADDR_STRLEN</span><span class="p">];</span>
<span class="n">ls</span> <span class="o">=</span> <span class="n">ngx_array_push</span><span class="p">(</span><span class="o">&</span><span class="n">cf</span><span class="o">-></span><span class="n">cycle</span><span class="o">-></span><span class="n">listening</span><span class="p">);</span>
<span class="n">ngx_memzero</span><span class="p">(</span><span class="n">ls</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">ngx_listening_t</span><span class="p">));</span>
<span class="n">sa</span> <span class="o">=</span> <span class="n">ngx_palloc</span><span class="p">(</span><span class="n">cf</span><span class="o">-></span><span class="n">pool</span><span class="p">,</span> <span class="n">socklen</span><span class="p">);</span>
<span class="n">ngx_memcpy</span><span class="p">(</span><span class="n">sa</span><span class="p">,</span> <span class="n">sockaddr</span><span class="p">,</span> <span class="n">socklen</span><span class="p">);</span>
<span class="n">ls</span><span class="o">-></span><span class="n">sockaddr</span> <span class="o">=</span> <span class="n">sa</span><span class="p">;</span>
<span class="n">len</span> <span class="o">=</span> <span class="n">ngx_sock_ntop</span><span class="p">(</span><span class="n">sa</span><span class="p">,</span> <span class="n">text</span><span class="p">,</span> <span class="n">NGX_SOCKADDR_STRLEN</span><span class="p">,</span> <span class="mi">1</span><span class="p">);</span>
<span class="n">ls</span><span class="o">-></span><span class="n">addr_text</span><span class="p">.</span><span class="n">data</span> <span class="o">=</span> <span class="n">ngx_pnalloc</span><span class="p">(</span><span class="n">cf</span><span class="o">-></span><span class="n">pool</span><span class="p">,</span> <span class="n">len</span><span class="p">);</span>
<span class="n">ngx_memcpy</span><span class="p">(</span><span class="n">ls</span><span class="o">-></span><span class="n">addr_text</span><span class="p">.</span><span class="n">data</span><span class="p">,</span> <span class="n">text</span><span class="p">,</span> <span class="n">len</span><span class="p">);</span>
<span class="p">...</span>
<span class="p">};</span>
</code></pre>
</div>
<p>在ngx_core.h中,加载了</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="cp">#include <ngx_socket.h>
</span></code></pre>
</div>
<p>所以结果就是说,nginx日志里记载的$remote_addr变量,就是由connection的socket里获得的。在socket.h里可以看到accept函数的定义:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="kt">int</span> <span class="n">accept</span><span class="p">(</span><span class="kt">int</span> <span class="n">sockfd</span><span class="p">,</span> <span class="kt">void</span> <span class="o">*</span><span class="n">addr</span><span class="p">,</span> <span class="kt">int</span> <span class="o">*</span><span class="n">addrlen</span><span class="p">);</span>
</code></pre>
</div>
<p>另外,nginx上除了$remote_addr变量外,还有一个$binary_remote_addr变量。而且在ngx_http_variables.c里,根据是否是IPv6协议,做了区分,最终地址是通过r->connection结构体里的sockaddr->sin_addr获得。</p>
<p>目前就看到这里了……关于socket如何从监听套接字上获得IP并建立连接套接字的,以后再继续研究TCP层上的知识。</p>
<h2 id="cookie-insert">2、cookie insert原理在负载均衡上是如何实现的。</h2>
<p>作7层负载均衡的时候,会遇到cookie类型的会话保持。</p>
<p>一般的session保持办法,是利用源地址哈希(source-hash)的办法,把同一个来源客户(实际通常是同一个C段的IP),固定指向后端的同一台机器。</p>
<p>而利用cookie的办法,则是在负载均衡器上,给响应客户请求的http-response-header里Set-Cookie字段添加上有关内容,然后根据客户请求的http-request-header里Cookie的该字段内容,分发到和之前一样的后端服务器上。</p>
<p>在nginx上没有标准模块完成这个事情,不过可以用<a href="http://wiki.nginx.org/HttpMapModule">map功能</a>进行简单的模拟,如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">map</span> <span class="nv">$COOKIE_route</span> <span class="nv">$group</span> <span class="p">{</span>
<span class="kn">700003508</span> <span class="s">admin</span><span class="p">;</span>
<span class="kn">~*3</span>$ <span class="s">admin</span><span class="p">;</span>
<span class="kn">default</span> <span class="s">user</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">upstream</span> <span class="s">backend_user</span> <span class="p">{</span>
<span class="kn">server</span> <span class="nf">10.3.24.11</span><span class="p">:</span><span class="mi">8080</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">upstream</span> <span class="s">backend_admin</span> <span class="p">{</span>
<span class="kn">server</span> <span class="nf">10.3.25.21</span><span class="p">:</span><span class="mi">8081</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">server</span> <span class="p">{</span>
<span class="kn">listen</span> <span class="mi">80</span><span class="p">;</span>
<span class="kn">server_name</span> <span class="s">photo.domain.com</span><span class="p">;</span>
<span class="kn">location</span> <span class="n">/</span> <span class="p">{</span>
<span class="kn">proxy_pass</span> <span class="s">http://backend_</span><span class="nv">$group</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>不过nginx社区有第三方模块叫做”nginx-sticky-module”的,用来完成这个功能。项目托管在googlecode上,具体地址是<a href="http://code.google.com/p/nginx-sticky-module">http://code.google.com/p/nginx-sticky-module</a>。具体实现的效果是首先根据轮训RR随机到某台后端,然后在响应的Set-Cookie上加上route=md5(upstream)字段,第二次请求再处理的时候,发现有route字段,直接导向原来的那台服务器。</p>
<p>编译后启用配置如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">upstream</span> <span class="p">{</span>
<span class="kn">sticky</span> <span class="s">[name=route]</span> <span class="s">[domain=.domain.com]</span> <span class="s">[path=/]</span> <span class="s">[expires=1h]</span> <span class="s">[hash=index|md5|sha1]</span> <span class="s">[no_fallback]</span><span class="p">;</span>
<span class="kn">server</span> <span class="nf">127.0.0.1</span><span class="p">:</span><span class="mi">9000</span><span class="p">;</span>
<span class="kn">server</span> <span class="nf">127.0.0.1</span><span class="p">:</span><span class="mi">9001</span><span class="p">;</span>
<span class="kn">server</span> <span class="nf">127.0.0.1</span><span class="p">:</span><span class="mi">9002</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
<h2 id="nginxworker80">3、nginx是多worker的,但是80端口只能有一个占用,这一段的工作原理是怎样的?</h2>
<p>这个问题的回答其实在第一个问题上已经部分涉及到了。就是socket的两个分类,一个是监听套接字,一个是连接套接字。占用80端口的,是使用的监听套接字。而worker里使用的,是accept之后建立的连接套接字。</p>
<p>正常情况下,nginx对worker加锁,在每一时刻,只有一个worker获得accept的权力。当监听的socket可以accept的时候,即有新链接时,主进程通过epoll的方式处理,先把这个事件保存起来,等通过锁的竞争选取一个worker后,再由这个worker真正的执行accept创建连接套接字,然后主进程返回监听状态。</p>
<p>代码中主要是ngx_trylock_accept_mutex()函数和ngx_process_events_and_timers()函数等,不过这个看不太懂,更多是根据别人的描述文章了。</p>
<h2 id="section">4、一致性哈希的原理。</h2>
<p>在7层负载均衡的时候,经常会利用到哈希。关于nginx上的url_hash和consistent_hash模块,我在2年前曾经简单的看过,博文链接如下:</p>
<ol>
<li><a href="http://chenlinux.com/2010/03/15/consistent_hash/">url_hash的perl脚本模拟</a></li>
<li><a href="http://chenlinux.com/2010/03/16/implement-consistent_hash-by-perl/">consistent_hash的perl脚本模拟</a></li>
</ol>
<p>两年后回头来看当初的脚本,真是很烂。不过从关键的uri和peer都取CRC32和取值做减法还是可以看出来一致性哈希的原理,即将节点通过哈希取值后均匀分布在一个0-9999999999的’圆环’上。然后要存储的url同样的算法取哈希值后,放进这个”圆环”里,顺时针方向离他最近的那个节点,即为他实际存储的节点。</p>
<p>在CPAN上,其实有<a href="http://search.cpan.org/~bradfitz/Set-ConsistentHash-0.92/lib/Set/ConsistentHash.pm">Set::ConsistentHash</a>模块可以看。如果是简单运用的话,<a href="http://search.cpan.org/~karavelov/Hash-ConsistentHash-0.05/lib/Hash/ConsistentHash.pm">Hash::ConsistentHash</a>模块是基于Set::ConsistentHash模块封装的易用版本。示例如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nn">Hash::</span><span class="nv">ConsistentHash</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">String::</span><span class="nv">CRC32</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$chash</span> <span class="o">=</span> <span class="nn">Hash::</span><span class="nv">ConsistentHash</span><span class="o">-></span><span class="k">new</span><span class="p">(</span> <span class="nv">buckets</span> <span class="o">=></span> <span class="p">{</span><span class="nv">A</span><span class="o">=></span><span class="mi">1</span><span class="p">,</span> <span class="nv">B</span><span class="o">=></span><span class="mi">2</span><span class="p">,</span> <span class="nv">C</span><span class="o">=></span><span class="mi">1</span><span class="p">},</span>
<span class="nv">hash_func</span><span class="o">=>\&</span><span class="nv">crc32</span><span class="p">,</span>
<span class="p">);</span>
<span class="k">my</span> <span class="nv">$server</span> <span class="o">=</span> <span class="nv">$chash</span><span class="o">-></span><span class="nv">get_bucket</span><span class="p">(</span><span class="s">'foo'</span><span class="p">);</span>
</code></pre>
</div>
<h2 id="inotify">5、inotify丢事件。</h2>
<p>这个问题没有碰到过,只在网上看到过一篇<a href="http://doc.chinaunix.net/linux/201007/687123.shtml">Linux事件监控机制遗漏事件问题的相关分析</a>,里面提到”发现在过于频繁的往目录下添加文件和目录的时候,会丢事件”。但是只提到了这么个问题,然后通过重复添加监听解决问题,没有提到原因。</p>
<p>我个人疑心,会不会是sysctl参数没有设置好的原因呢?</p>
<p>sysctl里关于inotify的参数有三个,如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">[</span>root@localhost ~]<span class="nv">$ </span>sudo /sbin/sysctl -a|grep inotify
fs.inotify.max_queued_events <span class="o">=</span> 16384
fs.inotify.max_user_watches <span class="o">=</span> 8192
fs.inotify.max_user_instances <span class="o">=</span> 128
</code></pre>
</div>
<p>上示是默认值,明显偏小。比方sersync2方案中,启动前就要求修改这些值到50000000。如果启动的时候在sysctl范围内,启动时没问题的,但是迅速的添加到了范围外,那么应该就会出这个问题了。</p>
<p>当然,以上是我个人猜测,也说不准真的是inotify本身却有问题。</p>
<h2 id="nginxworkercpu">7、nginx的worker是怎么绑定到cpu上的?</h2>
<p>nginx有一个配置,就是启动多个worker的时候,可以使用cpu_affinity配置将worker分别绑定在不同的cpu上。</p>
<p>如果有8个cpu,那么相应参数就是:</p>
<p>00000001 00000010 00000100 00001000 00010000 00100000 01000000 10000000</p>
<p>也就是类似占位符一样一个位置代表一个CPU。如果按照普通理解的二进制,那么0011不是第三个CPU而是绑定在第1和第2个CPU上平均……</p>
<p>这种写法,是由操作系统决定的。在nginx的ngx_process_cycle.c中,相关内容如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="cp">#include <ngx_config.h>
</span>
<span class="k">static</span> <span class="kt">void</span>
<span class="nf">ngx_worker_process_init</span><span class="p">(</span><span class="n">ngx_cycle_t</span> <span class="o">*</span><span class="n">cycle</span><span class="p">,</span> <span class="n">ngx_uint_t</span> <span class="n">priority</span><span class="p">)</span>
<span class="p">{</span>
<span class="p">...</span>
<span class="k">if</span> <span class="p">(</span><span class="n">cpu_affinity</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">sched_setaffinity</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="p">(</span><span class="n">cpu_set_t</span> <span class="o">*</span><span class="p">)</span> <span class="o">&</span><span class="n">cpu_affinity</span><span class="p">)</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
<span class="p">...</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>在ngx_config.h中:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="cp">#elif (NGX_LINUX)
#include <ngx_linux_config.h>
</span></code></pre>
</div>
<p>在ngx_linux_config.h中:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="cp">#include <sched.h>
</span></code></pre>
</div>
<p>其实可以直接通过man sched_setaffinity看说明:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="cp">#include <sched.h>
</span> <span class="kt">int</span> <span class="n">sched_setaffinity</span><span class="p">(</span><span class="n">pid_t</span> <span class="n">pid</span><span class="p">,</span> <span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">cpusetsize</span><span class="p">,</span>
<span class="n">cpu_set_t</span> <span class="o">*</span><span class="n">mask</span><span class="p">);</span>
</code></pre>
</div>
<p>关于这个*mask,man文档之后描述如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> The actual system
call interface is slightly different, with the mask being typed as unsigned long *, reflecting
that the fact that the underlying implementation of CPU sets is a simple bitmask.
</code></pre>
</div>
<p>bitmask就是上面说到的那个意思了~~</p>
<h2 id="section-1">8、某应用经过7层负载均衡访问应用服务器,因业务需要设置了5秒无响应即返回502错误。有反馈说全网范围内5%的访问出现错误,如何判断问题具体出在哪里?</h2>
<p>这个问题目前我还想不到有什么特别简捷的办法。靠类似nagios那样的定时监测,肯定是很不容易抓到错误的。如果靠debug日志或者strace命令啊,tcpdump命令啊的,在高流量的情况下,又太容易淹没在海量的正常数据里了。</p>
<p>另一个猜测是连接数满了,TCP的或者HTTP的。不过按理说负载均衡器上应该有监控,不至于到这么危急的时候还是通过客户端访问来反馈问题……</p>
perl函数返回值引起的误会
2012-03-04T00:00:00+08:00
perl
http://chenlinux.com/2012/03/04/magic-about-perl-subroutine-return-value
<p>在微博上偶然看到有位<a href="http://weibo.com/iheartbeat" title="南唐古韵">@南唐古韵</a>童鞋发了一条关于perl的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>发现perl的一个bug:(2**3)**2=8
</code></pre>
</div>
<p>显然perl不可能真的犯这么白痴的错误,那么问题在哪呢?我们先看看下面这个判断:<br />
<code class="highlighter-rouge">perl
[root@localhost ~]# perl -e 'print "OK" if (2**3)**2 == 8'
[root@localhost ~]# perl -e 'print "OK" if (2**3)**2 == 64'
OK
[root@localhost ~]#
</code><br />
一目了然,运算肯定是正确的。那上面那位童鞋的话是怎么得出来的呢?稍微想想,猜他可能是这样:<br />
<code class="highlighter-rouge">perl
[root@localhost ~]# perl -e 'print (2**3)**2'
8
</code><br />
哇,真的变成8啦?!</p>
<p>其实都是因为print搞的鬼啦~print作为内置的命令,我们在书写的时候经常用空格分隔开参数,而不记得其实标准的应该用中括号括起来的~<br />
也就是说,其实上面那行命令应该是:<br />
<code class="highlighter-rouge">perl
[root@localhost ~]# perl -e 'print(2**3) **2'
</code></p>
<p>然后下一个问题:在print之后还有一个**2啊,既然前面已经执行完成一个print命令,后面再加个东西,咋不会报错呢?</p>
<p>这就是涉及到关键了,perl的print执行也是有返回值的,真为1,假为0。也就是说,上面的命令其实是先执行了2的3次方得到8,然后执行print输出”8”到STDOUT并且返回1;然后执行1的2次方得到1;程序结束。</p>
<p>我们可以这样验证一下:<br />
<code class="highlighter-rouge">perl
[root@AY110907102215177d47d ~]# perl -e 'print (2**3)**2'
8[root@AY110907102215177d47d ~]# perl -e '$r=print (2**3)**2;print $r'
81[root@AY110907102215177d47d ~]# perl -e '$r=print (2**3)*2;print $r'
82[root@AY110907102215177d47d ~]# perl -e '$r=print (2**3)*2,"\n";print $r'
82[root@AY110907102215177d47d ~]# perl -e '$r=print (2**3,"\n")*2;print $r'
8
2[root@AY110907102215177d47d ~]#
</code><br />
因为要给print造一个返回值为假的示例不太容易,所以举一个1*2=2/0*2=0来证明咯~至于加上的”\n”测试,更进一步证明它跟print无关啦~~</p>
ZenLoadBalancer试用(一)
2012-02-29T00:00:00+08:00
http://chenlinux.com/2012/02/29/intro-zen-load-balancer
<p>在微博上看到一款叫做Zen LoadBalancer的负载均衡软件更新新闻。于是决定下载来看看,其官网地址是:<a href="http://www.zenloadbalancer.com/web/">http://www.zenloadbalancer.com/web/</a>。具体的说,是基于Debian构建的TCP/UDP负载均衡,通过perl的CGI页面完成对负载均衡的管理和监控,包括cluster功能(类似keepalived双机)。对后端real server的监控通过的是nagios-plugins里提供的check_http/check_smtp/check_ftp等一系列脚本完成的。监控通过SNMP完成,展示图像是rrd和GD::Graph。唯一没法知道的就是做负载均衡核心功能的pen这个东东,因为提供的是iso镜像,安装完后就直接有二进制文件在了,不知道这个pen到底是怎么做的……</p>
<h2 id="section">下载安装</h2>
<p>通过官网页面找到download即可,实际是托管在sf.net上。</p>
<p>安装步骤和普通的linux发行版光盘安装时一样的。作为实验,就只用电脑上的virtualbox挂载iso文件然后启动新虚拟机即可了。iso文件有225MB大,安装完成之后则是五百多兆,还是比较精简的了。</p>
<p>注意:在install的时候,会提示输入ip啊,gw啊之类的,如果有条件的话,最好就输正确的。因为在这步的时候zen就直接把输入结果写进zen配置文件而不是debian的/etc/network/interfaces了。然后每次重启zenloadbalancer都会根据zen的配置改变网络配置。</p>
<h2 id="section-1">项目路径和关键文件</h2>
<p>主要有两个启动脚本/etc/init.d/zenloadbalancer和/etc/init.d/mini_httpd,而zen的其他所有配置啊,执行文件啊,perl脚本啊,日志啊,都存放在/usr/local/zenloadbalancer目录下。</p>
<p>目录命名还算一目了然,主要有一个config和app比较混淆。</p>
<p>config里是用来存放负载均衡配置的地方,大多数情况下,这里的文件应该是通过webGUI自动生成的。包括if_<em>.conf指定网卡的(这也是唯一一个不全部由GUI生成的,如第一步所说,在install的时候会生成第一个eth0的配置),cluster_</em>.conf指定高可用集群的配置,*_farms.conf指定具体某个instance的配置(类似keepalived.conf里的一个virtual_server{}配置)。</p>
<p>app里包括了zen的主要应用,比如mini_httpd、pen、zeninotify等。其他的也没啥可看的,因为用不上改动。</p>
<p>pen里有man可以看,比较引人的就是它可以限制前后两段的链接数,包括单独设定到每个realserver的链接数。从启动脚本和man来看,应该就是类似lvs的NAT模式,不知道官网说的30000链接是社区版运行的结果,还是他们硬件版的结果~另一个比较怪异的事情是zen里带上了命令用来从realserver上获取日志,其man文档说是因为zen的负载均衡会传递给后端自己的IP,所以需要在zen收集原始ip,然后跟后端的日志合并?但是实际在运行的webGUI上看到明明有X-Forwarder-For支持。不知道是不是man没更新了?</p>
<p>mini_httpd是一个常用于嵌入式开发的webserver,支持HTTP/1.1协议,支持CGI,支持SSL。zen里就是用这个来发布其webGUI的。可以命令行参数启动,也可以写一个简单的配置文件。在我的测试中,默认配置文件启动是有问题的,每次连接都会被reset by peer。strace的结果显示mini_httpd进程fork出来的child总是异常退出。不过从配置文件里挑几个必要的参数写在命令行里启动就没问题了。</p>
<h2 id="webgui">webGUI介绍</h2>
<p>启动起来后,就可以登录操作的,默认采用了htpasswd控制访问权限,初始用户密码是admin/admin。首页是zenLB的监控图。</p>
<p>在farms的添加页里可以看到,负载均衡算法不多,就是轮训,哈希,权重,优先级。注意说明写的是每个clientIP做hash,而不是一般说的C段。而优先级的意思是:只在最高且相同优先级的server间均衡,除非都down掉了,才会指向低优先级的。</p>
<p>另外提供的功能还有:设定客户端保持时间,最大连接客户端数量,最大允许后端realserver数量,给httpheader加x-forwarder-for信息,启用nagios-plugins的外部监控等。</p>
<p>以上是tcp的配置,然后还可以具体的设置http/https的设置,注意https不是透传的,而是在zen上验证ssl,给后端的依然是http请求。</p>
<p>然后有zen自己的设置。主要有apt配置,可以添加apt源,包括zen自己的源,这样通过apt直接升级zenloadbalance。</p>
<p>然后有网卡设置。实际也就是通过ip addr命令添加咯。</p>
<p>然后有cluster设置。也就是在两台zenlb之间,通过ssh证书信任,共同使用另一个vip服务。也有master-slave抢占和slave-slave不抢占两种运行模式。两台zenlb之间,通过zenlate</p>
<p>ncy命令的UCARP服务来保持心跳,通过zeninotify传输master上的配置到slave。</p>
<p>然后还有日志、配置备份等功能……</p>
<p>页面上,还有一个和cluster VIP并列的选项是Vlan IP添加。但是在官网的指南上没看到相关内容,我点击后也没出现什么有意思的结果,怀疑会不会是未完成的功能。</p>
<p>基本上就是这样。手头没有机器给我折腾,只能笔记本电脑上解解眼馋了…</p>
从wordpress博客迁移到github记录
2012-02-14T00:00:00+08:00
perl
http://chenlinux.com/2012/02/14/move-wordpress-to-github
<p>因为原先托管wordpress博客的阿里云主机被通管局要求备案,加上通管局早令夕改的浪费我来回的快递费,于是决定搬迁到国外,github page的免费托管就此进入我的视界。</p>
<h2 id="github">第一步,申请github账号</h2>
<p>这步大同小异,想个好名字就是了。</p>
<h2 id="section">第二步,创建新项目</h2>
<p>也就是注册完毕后在页面上看到的<a href="https://github.com/repositories/new">“New repository”</a></p>
<p>在页面上填上项目名字。注意如果是要做博客的话,命名是有规范的,必须是yourusername.github.com这样子。</p>
<h2 id="git">第三步,设置本地git环境</h2>
<p>这步参见[“help页面”]:(http://help.github.com/win-set-up-git/)</p>
<h2 id="rubyjekyll">第四步,配置ruby和jekyll环境</h2>
<p>一般来说,linux设备上肯定都已经有了ruby,不过版本比较低,所以升级ruby再使用:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>
<span class="o">[</span>root@localhost ~]# wget https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer
<span class="o">[</span>root@localhost ~]# bash rvm-installer
<span class="o">[</span>root@localhost ~]# rvm install 1.9.3
<span class="o">[</span>root@localhost ~]# rvm --default 1.9.3
<span class="o">[</span>root@localhost ~]# gem install jekyll
</code></pre>
</div>
<p>这样就可以在本地运行jekyll查看博客效果了。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="o">[</span>root@localhost youusername.github.com]# jekyll --server
</code></pre>
</div>
<h2 id="section-1">第五步,创建本地博客目录</h2>
<p>这步,可以直接fork已有的github博客代码来用,比如:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>
<span class="o">[</span>root@localhost ~]# git clone https://github.com/plusjade/jekyll-bootstrap.git yourusername.github.com
<span class="o">[</span>root@localhost ~]# <span class="nb">cd </span>yourusername.github.com
<span class="o">[</span>root@localhost yourusername.github.com]# git remote <span class="nb">set</span>-url origin git@github.com:yourusername/yourusername.github.com.git
<span class="o">[</span>root@localhost yourusername.github.com]# git add .
<span class="o">[</span>root@localhost yourusername.github.com]# git commit -m <span class="s1">'first commit'</span>
<span class="o">[</span>root@localhost yourusername.github.com]# git push origin master
</code></pre>
</div>
<h2 id="section-2">第六步,修改必要信息</h2>
<p>最重要的一步,创建目录下的CNAME信息,如果是fork的其他人的项目,那么修改CNAME的内容为自己的域名,比如blog.yourdomain.me。不然只能通过github的二级域名yourusername.github.com来访问了。<br />
然后跟github无关的重要一步,上godaddy或者dnspod之类的地方,修改NS记录。注意这里,虽然上面文件叫CNAME,dns上却是要配置一个A记录,把A记录指向207.97.227.245即可。</p>
<h2 id="section-3">第七步,添加评论功能</h2>
<p>因为github提供的是一个静态页面托管,所以评论功能是需要用其他地方提供的。大家都用的是disqus的插件。所以上<a href="http://disqus.com/">disqus主页</a> 去申请一个账号吧。然后按照提示,将提供的插件代码键入_includes/post.html中—-不过一般来说,fork出来的都已经有了。</p>
<p>2012-03-16 补充:<br />
最近也有其他的社会化第三方评论系统出现,可以使用微博、QQ来登录评论,免去上disqus注册之苦。大家可以上<a href="http://uyan.cc/getcode?index">友言</a>看看。</p>
<p>2012-04-29 补充:<br />
今天收到友言的提示邮件,上去看了一下,其实友言是支持<a href="http://uyan.cc/setting/backup">xml导入评论</a>的。那么我们就可以把wp里的评论也一块导出来玩了:<br />
```perl</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/perl</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">DBI</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">DBD::</span><span class="nv">mysql</span><span class="p">;</span>
<span class="nb">open</span> <span class="k">my</span> <span class="nv">$fh</span><span class="p">,</span> <span class="s">'>'</span><span class="p">,</span> <span class="s">'comments.xml'</span> <span class="ow">or</span> <span class="nb">die</span> <span class="vg">$!</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">$fh</span> <span class="s">'<?xml version="1.0" encoding="utf-8"?>'</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">$fh</span> <span class="s">'<uyan xmlns="http://uyan.cc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$dbh</span> <span class="o">=</span> <span class="nv">DBI</span><span class="o">-></span><span class="nb">connect</span><span class="p">(</span><span class="s">"DBI:mysql:database=blog;host=localhost;port=3306"</span><span class="p">,</span> <span class="s">"user"</span><span class="p">,</span> <span class="s">"passwd"</span><span class="p">,</span> <span class="p">{</span><span class="nv">RaiseError</span> <span class="o">=></span> <span class="mi">1</span><span class="p">});</span>
<span class="nv">$dbh</span><span class="o">-></span><span class="k">do</span><span class="p">(</span><span class="s">'set names utf8'</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$sth</span> <span class="o">=</span> <span class="nv">$dbh</span><span class="o">-></span><span class="nv">prepare</span><span class="p">(</span><span class="s">'select p.post_title title, c.comment_author author, c.comment_author_url url, c.comment_date date, c
.comment_content content from wp_comments c, wp_posts p where c.comment_approved='</span><span class="mi">1</span><span class="s">' and c.comment_post_ID = p.ID'</span><span class="p">);</span>
<span class="nv">$sth</span><span class="o">-></span><span class="nv">execute</span><span class="p">();</span>
<span class="k">while</span> <span class="p">(</span><span class="k">my</span> <span class="nv">$ref</span> <span class="o">=</span> <span class="nv">$sth</span><span class="o">-></span><span class="nv">fetchrow_hashref</span><span class="p">())</span> <span class="p">{</span>
<span class="k">print</span> <span class="nv">$fh</span> <span class="s">'<post><content>'</span><span class="o">.</span><span class="nv">$ref</span><span class="o">-></span><span class="p">{</span><span class="s">'content'</span><span class="p">}</span><span class="o">.</span><span class="s">'</content>'</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">$fh</span> <span class="s">'<page_title>'</span><span class="o">.</span><span class="nv">$ref</span><span class="o">-></span><span class="p">{</span><span class="s">'title'</span><span class="p">}</span><span class="o">.</span><span class="s">'</page_title>'</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">$fh</span> <span class="s">'<user_id>0</user_id>'</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">$fh</span> <span class="s">'<comment_author>'</span><span class="o">.</span><span class="nv">$ref</span><span class="o">-></span><span class="p">{</span><span class="s">'author'</span><span class="p">}</span><span class="o">.</span><span class="s">'</comment_author>'</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">$fh</span> <span class="s">'<time>'</span><span class="o">.</span><span class="nv">$ref</span><span class="o">-></span><span class="p">{</span><span class="s">'date'</span><span class="p">}</span><span class="o">.</span><span class="s">'</time>'</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">$fh</span> <span class="s">'<from_type>wordpress</from_type>'</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">$fh</span> <span class="s">'<page>'</span><span class="o">.</span><span class="nv">$ref</span><span class="o">-></span><span class="p">{</span><span class="s">'url'</span><span class="p">}</span><span class="o">.</span><span class="s">'<page>'</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">$fh</span> <span class="s">'<page_url>'</span><span class="o">.</span><span class="nv">$ref</span><span class="o">-></span><span class="p">{</span><span class="s">'url'</span><span class="p">}</span><span class="o">.</span><span class="s">'</page_url>'</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">$fh</span> <span class="s">'<domain>youdomain.com</domain></post>'</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">print</span> <span class="nv">$fh</span> <span class="s">'</uyan>'</span><span class="p">;</span>
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>
## 第八步,导出wordpress博客文章
jekyll提供了ruby脚本/usr/local/rvm/gems/ruby-1.9.3-p0/gems/jekyll-0.11.2/lib/jekyll/migrators/wordpress.rb来专门做这件事情,只需要运行如下命令即可:
```bash
[root@localhost yourusername.github.com]# gem install sequel
[root@localhost yourusername.github.com]# gem install mysql -- --with-mysql-config=/usr/bin/mysql_config
[root@localhost yourusername.github.com]# ruby -rubygems -e 'require "jekyll/migrators/wordpress"; Jekyll::WordPress.process("database", "user", "pass")'
</code></pre>
</div>
<p>不过我这里运行有点问题,ruby又不太懂,再看完上面这个脚本的意思后,干脆放弃排错,直接用perl完成了如下这个脚本:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>
<span class="c1">#!env perl</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">DBI</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">DBD::</span><span class="nv">mysql</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$dbh</span> <span class="o">=</span> <span class="nv">DBI</span><span class="o">-></span><span class="nb">connect</span><span class="p">(</span><span class="s">"DBI:mysql:database=blog;host=localhost;port=3306"</span><span class="p">,</span> <span class="s">"user"</span><span class="p">,</span> <span class="s">"passwd"</span><span class="p">,</span> <span class="p">{</span><span class="nv">RaiseError</span> <span class="o">=></span> <span class="mi">1</span><span class="p">});</span>
<span class="nv">$dbh</span><span class="o">-></span><span class="k">do</span><span class="p">(</span><span class="s">'set names utf8'</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$sth</span> <span class="o">=</span> <span class="nv">$dbh</span><span class="o">-></span><span class="nv">prepare</span><span class="p">(</span><span class="s">"select post_title, post_name, post_date, post_content FROM wp_posts WHERE post_status = 'publish' AND post_type = 'post' order by id desc "</span><span class="p">);</span>
<span class="nv">$sth</span><span class="o">-></span><span class="nv">execute</span><span class="p">();</span>
<span class="k">while</span> <span class="p">(</span><span class="k">my</span> <span class="nv">$ref</span> <span class="o">=</span> <span class="nv">$sth</span><span class="o">-></span><span class="nv">fetchrow_hashref</span><span class="p">())</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$slug</span> <span class="o">=</span> <span class="nv">$ref</span><span class="o">-></span><span class="p">{</span><span class="s">'post_name'</span><span class="p">};</span>
<span class="k">next</span> <span class="k">unless</span> <span class="nv">$slug</span> <span class="o">=~</span> <span class="sr">m/^\w/</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$title</span> <span class="o">=</span> <span class="nv">$ref</span><span class="o">-></span><span class="p">{</span><span class="s">'post_title'</span><span class="p">};</span>
<span class="k">my</span> <span class="nv">$date</span> <span class="o">=</span> <span class="nb">substr</span><span class="p">(</span><span class="nv">$ref</span><span class="o">-></span><span class="p">{</span><span class="s">'post_date'</span><span class="p">},</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">10</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$content</span> <span class="o">=</span> <span class="nv">$ref</span><span class="o">-></span><span class="p">{</span><span class="s">'post_content'</span><span class="p">};</span>
<span class="k">my</span> <span class="nv">$filename</span> <span class="o">=</span> <span class="nv">$date</span> <span class="o">.</span> <span class="s">'-'</span> <span class="o">.</span> <span class="nv">$slug</span> <span class="o">.</span> <span class="s">'.textile'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$header</span> <span class="o">=</span> <span class="s">"---\nlayout: post\ntitle: "</span> <span class="o">.</span> <span class="nv">$title</span> <span class="o">.</span> <span class="s">"\ndate: "</span> <span class="o">.</span> <span class="nv">$date</span> <span class="o">.</span> <span class="s">"\n---\n\n"</span><span class="p">;</span>
<span class="nb">open</span> <span class="k">my</span> <span class="nv">$fh</span><span class="p">,</span> <span class="s">'>'</span><span class="p">,</span> <span class="s">"$filename"</span> <span class="ow">or</span> <span class="nb">die</span> <span class="vg">$!</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">$fh</span> <span class="nv">$header</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">$fh</span> <span class="s">"$content\n"</span><span class="p">;</span>
<span class="nv">$fh</span><span class="o">-></span><span class="nb">close</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">$sth</span><span class="o">-></span><span class="nv">finish</span><span class="p">();</span>
<span class="nv">$dbh</span><span class="o">-></span><span class="nv">disconnect</span><span class="p">();</span>
</code></pre>
</div>
<p>注意到中文url的不可读性,需要提前把wordpress的博客静态化url改成英文的。好在我一年前已经这么做了。<br />
比较郁闷的一点是:wordpress的tag和category存放的表结构太恶心了,犹豫很久,最后放弃了导出博文对应category和tags的想法…</p>
dotcloud试用
2012-02-13T00:00:00+08:00
cloud
perl
http://chenlinux.com/2012/02/13/using-perl-on-dotcloud
<p>dotcloud是日本的一个PAAS厂商。一年多前因为plack作者的加入推出了对perl的支持。我这几个月没事做,今天想起来试试看。用起来确实蛮舒服的。(注:大部分内容是网上已经有了的,我这只是记录一下自己的步骤)<br />
## 第一步,申请账号</p>
<p>就跟普通的网站注册一样,填用户名密码邮箱,邮箱邮件激活——完毕。</p>
<h2 id="section">第二步,安装客户端</h2>
<p>跟安装其他python模块一样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">easy_install</span> <span class="n">pip</span> <span class="o">&&</span> <span class="n">pip</span> <span class="n">install</span> <span class="n">dotcloud</span>
</code></pre>
</div>
<h2 id="section-1">第三步,个人密钥认证</h2>
<p>在dotcloud的个人主页( “settings”:http://www.dotcloud.com/account/settings )上就能看到个人密钥。然后在终端里输入密钥:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">[</span>root@localhost ~]# dotcloud
Enter your api key:
</code></pre>
</div>
<p>这个密钥就存在了~/.dotcloud/dotcloud.conf里,以后就不用再认证了。</p>
<h2 id="section-2">第四步,创建项目文件</h2>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">[</span>root@localhost ~]# mkdir myapp-on-dotcloud <span class="o">&&</span> <span class="nb">cd </span>myapp-on-dotcloud
<span class="o">[</span>root@localhost myapp-on-dotcloud]# dotcloud create myapp-on-dotcloud
<span class="o">[</span>root@localhost myapp-on-dotcloud]# dancer -a helloworld
<span class="o">[</span>root@localhost myapp-on-dotcloud]# touch dotcloud.yml
<span class="o">[</span>root@localhost myapp-on-dotcloud]# <span class="nb">echo</span> <span class="s2">"require 'bin/app.pl';"</span> > helloworld/app.psgi
</code></pre>
</div>
<p>用dotcloud命令创建项目myapp-on-dotcloud,并且在项目中运用dancer。唯一需要多加一个文件app.psgi,这个文件是云环境中psgi运行时需要读取的。</p>
<h2 id="section-3">修改云环境配置</h2>
<ul>
<li>dotcloud.yml的配置,这相当于云环境的Basic File:</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="s">www</span><span class="pi">:</span>
<span class="s">type</span><span class="pi">:</span> <span class="s">perl</span>
<span class="s">approot</span><span class="pi">:</span> <span class="s">helloworld</span>
<span class="s">requirements</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">Template::Toolkit</span>
<span class="pi">-</span> <span class="s">JSON</span>
<span class="s">db</span><span class="pi">:</span>
<span class="s">type</span><span class="pi">:</span> <span class="s">mysql</span>
</code></pre>
</div>
<p>这个yaml文件,第一级是节点的名字,可以随意取名,主要的是type,必须是dotcloud支持的,比如静态文件的static,动态应用的perl,数据库的mysql等等。然后是approot,指定web应用的/路径。最后是requirements,不过这个也可以通过Makefile.PL文件来指明。</p>
<ul>
<li>Makefile.PL的修改:</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">ExtUtils::</span><span class="nv">MakeMaker</span><span class="p">;</span>
<span class="nv">WriteMakefile</span><span class="p">(</span>
<span class="nv">NAME</span> <span class="o">=></span> <span class="s">'helloworld'</span><span class="p">,</span>
<span class="nv">AUTHOR</span> <span class="o">=></span> <span class="sx">q{YOUR NAME <youremail@example.com>}</span><span class="p">,</span>
<span class="nv">VERSION_FROM</span> <span class="o">=></span> <span class="s">'lib/helloworld.pm'</span><span class="p">,</span>
<span class="nv">ABSTRACT</span> <span class="o">=></span> <span class="s">'YOUR APPLICATION ABSTRACT'</span><span class="p">,</span>
<span class="p">(</span><span class="nv">$</span><span class="nn">ExtUtils::MakeMaker::</span><span class="nv">VERSION</span> <span class="o">>=</span> <span class="mf">6.3002</span>
<span class="p">?</span> <span class="p">(</span><span class="s">'LICENSE'</span><span class="o">=></span> <span class="s">'perl'</span><span class="p">)</span>
<span class="p">:</span> <span class="p">()),</span>
<span class="nv">PL_FILES</span> <span class="o">=></span> <span class="p">{},</span>
<span class="nv">PREREQ_PM</span> <span class="o">=></span> <span class="p">{</span>
<span class="s">'Test::More'</span> <span class="o">=></span> <span class="mi">0</span><span class="p">,</span>
<span class="s">'YAML'</span> <span class="o">=></span> <span class="mi">0</span><span class="p">,</span>
<span class="s">'Dancer'</span> <span class="o">=></span> <span class="mf">1.3080</span><span class="p">,</span>
<span class="s">'Plack'</span> <span class="o">=></span> <span class="mf">0.9985</span><span class="p">,</span>
<span class="p">},</span>
<span class="nv">dist</span> <span class="o">=></span> <span class="p">{</span> <span class="nv">COMPRESS</span> <span class="o">=></span> <span class="s">'gzip -9f'</span><span class="p">,</span> <span class="nv">SUFFIX</span> <span class="o">=></span> <span class="s">'gz'</span><span class="p">,</span> <span class="p">},</span>
<span class="nv">clean</span> <span class="o">=></span> <span class="p">{</span> <span class="nv">FILES</span> <span class="o">=></span> <span class="s">'helloworld-*'</span> <span class="p">},</span>
<span class="p">);</span>
</code></pre>
</div>
<p>这个文件绝大多数是dancer命令自动创建的。唯一需要补充的一行是Plack。因为dotcloud不能自动根据dancer安装plack环境。</p>
<h2 id="section-4">第六步,上传项目,等待环境部署</h2>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">[</span>root@localhost myapp-on-dotcloud]# dotcloud push myapp-on-dotcloud
</code></pre>
</div>
<p>然后可以看到程序首先是启动rsync命令,根据UNIX时间戳比对文件变化。上传新文件后,会根据最新的Makefile的配置安装相应的模块。最后初始化项目,创建相应的请求路由。</p>
<h2 id="section-5">第七步,检查环境数据</h2>
<p>运行dotcloud info myapp-on-dotcloud命令,即可看到如下输出:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="s">db</span><span class="pi">:</span>
<span class="s">config</span><span class="pi">:</span>
<span class="s">mysql_masterslave</span><span class="pi">:</span> <span class="s">true</span>
<span class="s">mysql_password</span><span class="pi">:</span> <span class="s">A1BAAaAaaaA5Aa1aAAAa</span>
<span class="s">instances</span><span class="pi">:</span> <span class="s">1</span>
<span class="s">type</span><span class="pi">:</span> <span class="s">mysql</span>
<span class="s">www</span><span class="pi">:</span>
<span class="s">config</span><span class="pi">:</span>
<span class="s">path</span><span class="pi">:</span> <span class="s">/</span>
<span class="s">plack_env</span><span class="pi">:</span> <span class="s">deployment</span>
<span class="s">static</span><span class="pi">:</span> <span class="s">static</span>
<span class="s">uwsgi_processes</span><span class="pi">:</span> <span class="s">4</span>
<span class="s">instances</span><span class="pi">:</span> <span class="s">1</span>
<span class="s">type</span><span class="pi">:</span> <span class="s">perl</span>
<span class="s">url</span><span class="pi">:</span> <span class="s">http://myapp-on-dotcloud-user.dotcloud.com/</span>
</code></pre>
</div>
<p>于是我们就可以看到数据库的密码了。然后我们可以这样运用dotcloud的数据库:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">[</span>root@localhost helloworld]# dotcloud run myapp-on-dotcloud.db -- mysql -uroot -pA1BAAaAaaaA5Aa1aAAAa
<span class="c"># mysql -uroot -pA1BAAaAaaaA5Aa1aAAAa</span>
Welcome to the MySQL monitor. Commands end with ; or <span class="se">\g</span>.
Your MySQL connection id is 34
Server version: 5.1.41-3ubuntu12.10-log <span class="o">(</span>Ubuntu<span class="o">)</span>
Type <span class="s1">'help;'</span> or <span class="s1">'\h'</span> <span class="k">for </span>help. Type <span class="s1">'\c'</span> to clear the current input statement.
<span class="gp">mysql> </span>show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
+--------------------+
2 rows <span class="k">in </span><span class="nb">set</span> <span class="o">(</span>0.00 sec<span class="o">)</span>
<span class="gp">mysql> </span>Bye
</code></pre>
</div>
<p>可以看到,创建出来的mysql节点,是带有replica功能的,而且默认没有创建项目同名的库,需要自己创建。<br />
类似的,也可以通过ssh管理项目节点。详细的查看命令是dotcloud info raocl.www(看起来很面向对象化吧):</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="s">aliases</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">myapp-on-dotcloud-user.dotcloud.com</span>
<span class="s">build_revision</span><span class="pi">:</span> <span class="s">rsync-1329106583210</span>
<span class="s">config</span><span class="pi">:</span>
<span class="s">path</span><span class="pi">:</span> <span class="s">/</span>
<span class="s">plack_env</span><span class="pi">:</span> <span class="s">deployment</span>
<span class="s">static</span><span class="pi">:</span> <span class="s">static</span>
<span class="s">uwsgi_processes</span><span class="pi">:</span> <span class="s">4</span>
<span class="s">created_at</span><span class="pi">:</span> <span class="s">1329103902.969918</span>
<span class="s">datacenter</span><span class="pi">:</span> <span class="s">Amazon-us-east-1a</span>
<span class="s">image_version</span><span class="pi">:</span> <span class="s">87ce0731fd95 (latest)</span>
<span class="s">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">name</span><span class="pi">:</span> <span class="s">ssh</span>
<span class="s">url</span><span class="pi">:</span> <span class="s">ssh://dotcloud@myapp-on-dotcloud-user.dotcloud.com:7478</span>
<span class="pi">-</span> <span class="s">name</span><span class="pi">:</span> <span class="s">http</span>
<span class="s">url</span><span class="pi">:</span> <span class="s">http://myapp-on-dotcloud-user.dotcloud.com/</span>
<span class="s">state</span><span class="pi">:</span> <span class="s">running</span>
<span class="s">type</span><span class="pi">:</span> <span class="s">perl</span>
</code></pre>
</div>
<p>这里可以清楚的看到节点是运行在amazon的EC2上的,开起来7478端口的ssh可用。当然没必要自己用ssh去链接,因为可以这样直接运行:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">[</span>root@localhost helloworld]# dotcloud ssh myapp-on-dotcloud.www
<span class="c"># $SHELL</span>
<span class="gp">dotcloud@myapp-on-dotcloud-default-www-0:~$ </span>id
<span class="nv">uid</span><span class="o">=</span>1000<span class="o">(</span>dotcloud<span class="o">)</span> <span class="nv">gid</span><span class="o">=</span>33<span class="o">(</span>www-data<span class="o">)</span> <span class="nv">groups</span><span class="o">=</span>33<span class="o">(</span>www-data<span class="o">)</span>
<span class="gp">dotcloud@myapp-on-dotcloud-default-www-0:~$ </span>w
06:36:34 up 62 days, 21:20, 1 user, load average: 1.26, 1.52, 2.07
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
<span class="gp">dotcloud@myapp-on-dotcloud-default-www-0:~$ </span>ps auxwwf
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
dotcloud 189 0.0 0.0 70632 1556 ? S 06:35 0:00 sshd: dotcloud@pts/0
dotcloud 190 0.0 0.0 19416 2100 pts/0 Ss 06:35 0:00 <span class="se">\_</span> /bin/bash
dotcloud 204 0.0 0.0 15292 1148 pts/0 R+ 06:36 0:00 <span class="se">\_</span> ps auxwwf
dotcloud 146 0.0 0.0 42016 11608 ? Ss 04:17 0:01 /usr/bin/python /usr/bin/supervisord.real
dotcloud 153 0.0 0.0 73500 22516 ? S 04:17 0:00 <span class="se">\_</span> /usr/local/bin/uwsgi --pidfile /var/dotcloud/uwsgi.pid -s /var/dotcloud/uwsgi.sock --chmod-socket<span class="o">=</span>660 --master --processes 4 --psgi app.psgi --disable-logging
dotcloud 165 0.0 0.0 73500 19728 ? S 04:17 0:00 <span class="se">\_</span> /usr/local/bin/uwsgi --pidfile /var/dotcloud/uwsgi.pid -s /var/dotcloud/uwsgi.sock --chmod-socket<span class="o">=</span>660 --master --processes 4 --psgi app.psgi --disable-logging
dotcloud 166 0.0 0.0 73500 19728 ? S 04:17 0:00 <span class="se">\_</span> /usr/local/bin/uwsgi --pidfile /var/dotcloud/uwsgi.pid -s /var/dotcloud/uwsgi.sock --chmod-socket<span class="o">=</span>660 --master --processes 4 --psgi app.psgi --disable-logging
dotcloud 167 0.0 0.0 73500 19728 ? S 04:17 0:00 <span class="se">\_</span> /usr/local/bin/uwsgi --pidfile /var/dotcloud/uwsgi.pid -s /var/dotcloud/uwsgi.sock --chmod-socket<span class="o">=</span>660 --master --processes 4 --psgi app.psgi --disable-logging
dotcloud 168 0.0 0.0 73500 19728 ? S 04:17 0:00 <span class="se">\_</span> /usr/local/bin/uwsgi --pidfile /var/dotcloud/uwsgi.pid -s /var/dotcloud/uwsgi.sock --chmod-socket<span class="o">=</span>660 --master --processes 4 --psgi app.psgi --disable-logging
</code></pre>
</div>
<p>这下看到了,其实就是用uwsgi运行psgi程序。题外话:发现用的是supervisord做进程管理。<br />
这个时候就可以通过http://myapp-on-dotcloud.dotcloud.com/访问到dancer的index页面了~熟悉的dancing。。。。可以看到,整个dotcloud环境是比较接近server环境的,除了上传的几个特殊文件以外,基本跟普通的dancer开发web一样。</p>
OMD系列(四)mod_gearman配置与运行
2011-12-27T00:00:00+08:00
monitor
nagios
gearman
http://chenlinux.com/2011/12/27/conf_run_mod_gearman
<p>上一篇提到了shinken,这是一个完全重写过的nagios-like系统,如果想尽可能的在原有nagios知识基础上进行高性能的分布式扩展,那么可以使用mod_gearman模块。这个模块可以从github.com上直接获取使用,地址是:<br />
<a href="https://github.com/sni/mod_gearman">https://github.com/sni/mod_gearman</a><br />
当然,我们这里依然是直接采用了OMD分发的方式,事实上github上也是建议直接采用omd部署,包括升级也可以直接omd update~~(因为从源代码编译的话,在nagios3.2.2版本之前还需要打个patch才能支持eventhandler,既然都得重编译,直接搞新的得了)<br />
在github的“How it works”和“Common Scenarios”两章节里,详细的用图例说明了mod_gearman的工作原理和各种配置情形。我这里就不重复贴了。大概的说,就是可以把nagios配置检测项中的hosts/services/eventhandlers/hostgroups/servicegroups都单独成队列,然后启动多个worker有针对性的完成队列里的任务,最后也是用单独的result队列回收检测结果。<br />
嗯,从原理上来讲,hosts/services/eventhandlers的队列,属于load balance范围,而hostgroups/servicegroups的队列,属于distribute范围。<br />
(mod_gearman还有send_gearman程序用来load balance接收NSCA分布式监控程序的数据,这里就不说了)</p>
<hr />
<p>概念讲完了,现在说配置。<br />
要启用mod_gearman,方法和上篇启用shinken一样简单,运行omd config命令,在Distribute选项中选择Mod_gearman为on,然后omd start即可。<br />
注意,omd分发中只带了mod_gearman.so和client/worker/init脚本,你必须自己yum install gearmand安装jobserver才行。<br />
在配置完成重启动的时候,OMD就会自动的修改nagios.cfg里的broker_module配置为:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">broker_module</span><span class="o">=</span>.../mod_gearman.o <span class="nv">server</span><span class="o">=</span>localhost:4730 <span class="nv">eventhandler</span><span class="o">=</span>yes <span class="nv">services</span><span class="o">=</span>yes <span class="nv">hosts</span><span class="o">=</span>yes
</code></pre>
</div>
<p>同时启动1个</p>
<div class="highlighter-rouge"><pre class="highlight"><code> /omd/sites/monitor/version/sbin/gearmand --port<span class="o">=</span>4730 --pid-file<span class="o">=</span>/omd/sites/monitor/tmp/run/gearmand.pid --daemon --job-retries<span class="o">=</span>0 --threads<span class="o">=</span>10 --log-file<span class="o">=</span>/omd/sites/monitor/var/log/gearman/gearmand.log --verbose<span class="o">=</span>2 --listen<span class="o">=</span>localhost
</code></pre>
</div>
<p>老文章里写的gearmand默认端口都是7003,因为跟AFS冲突,所以现在的版本默认都是4730了;<br />
同时启动3个</p>
<div class="highlighter-rouge"><pre class="highlight"><code>/omd/sites/monitor/bin/mod_gearman_worker -d --config<span class="o">=</span>/omd/sites/monitor/etc/mod-gearman/worker.cfg --pidfile<span class="o">=</span>/omd/sites/monitor/tmp/run/gearman_worker.pid
</code></pre>
</div>
<p>OK,现在这5个worker,会由gearmand平均分配hosts/services/eventhandlers任务。一个基础的load balance就完成了。</p>
<hr />
<p>下一步就是前面命令里用到了的~/etc/mod-gearman/worker.cfg配置文件了。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">debug</span><span class="o">=</span>0
<span class="nv">config</span><span class="o">=</span>/omd/sites/monitor/etc/mod-gearman/port.conf
<span class="nv">eventhandler</span><span class="o">=</span>yes
<span class="nv">services</span><span class="o">=</span>yes
<span class="nv">hosts</span><span class="o">=</span>yes
<span class="c">#hostgroups=name2,name3</span>
<span class="c">#servicegroups=name1,name2,name3</span>
<span class="nv">encryption</span><span class="o">=</span>yes
<span class="nv">keyfile</span><span class="o">=</span>/omd/sites/monitor/etc/mod-gearman/secret.key
<span class="nv">logfile</span><span class="o">=</span>/omd/sites/monitor/var/log/gearman/worker.log
min-worker<span class="o">=</span>3
max-worker<span class="o">=</span>50
idle-timeout<span class="o">=</span>10
max-jobs<span class="o">=</span>500
spawn-rate<span class="o">=</span>1
<span class="nv">fork_on_exec</span><span class="o">=</span>no
<span class="nv">show_error_output</span><span class="o">=</span>yes
<span class="nv">workaround_rc_25</span><span class="o">=</span>off
</code></pre>
</div>
<p>以上是默认生成的配置文件。主要是定义最小的worker数量,最大的worker数量,最多运行的任务数量,等待超时时间,有等待任务时派生新worker的速度,是否为每个插件采用fork方式执行(这里注释写的是默认yes,但是OMD生成配置默认却是no,不知为何);另一部分配置就是关于lb和dist的了:hosts/services/eventhandlers的yes/no控制是否lb,hostgroups/servicegroups控制dist哪些group到具体的worker上。<br />
假设我们现在有3个servicegroup,分别叫name1/name2/name3,那么开启选项后运行omd reload命令。查看gearmand的状态如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> OMD[monitor]:~<span class="nv">$ </span>telnet 127.0.0.1 4730
Trying 127.0.0.1...
Connected to localhost <span class="o">(</span>127.0.0.1<span class="o">)</span>.
Escape character is <span class="s1">'^]'</span>.
status
check_results 0 0 1
worker_localhost 0 0 1
servicegroup_name1 0 0 3
servicegroup_name2 0 0 3
servicegroup_name3 0 0 3
host 0 0 3
service 0 0 3
eventhandler 0 0 3
dummy 0 0 5
.
</code></pre>
</div>
<p>表示有3个worker注册在gearman的这几个任务下了。<br />
(如果不用OMD,那么上面这些修改配置,分别启动worker指定不同配置等等动作,都要自己完成,参见github里的Installation章节From Source部份)</p>
OMD系列(三)shinken的discovery配置与运行
2011-12-20T00:00:00+08:00
monitor
nagios
shinken
http://chenlinux.com/2011/12/20/shinken_discovery_runner
<p>上篇说到OMD的可以通过omd config选择core,有nagios和shinken两个选项。shinken是一个完全重写的监控系统,但是在外部接口上,又是nagios-like的,甚至可以单纯的只用shinken的WebUI给nagios使。所以OMD作为nagios周边的项目,也就把shinken加入到core选项里了。<br />
不过经过试验发现,虽然omd的rpm是针对centos5.5的,代码中调用的有些命令版本却比较高,比如nmap命令使用了–traceroute选项,查ChangeLog发现是4.76版才加上的,而centos5上面的还是4.11版,所以要重新安装:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget http://nmap.org/dist/nmap-5.51-1.x86_64.rpm
yum remove nmap
yum install --nogpgcheck nmap-5.51-1.x86_64.rpm
</code></pre>
</div>
<p>又比如python,centos5默认安装的是2.4版,而nmap_parser中调用了2.5版才有的xml.tree.elementtree模块,所以也要升级。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget http://www.python.org/ftp/python/2.7.2/Python-2.7.2.tgz
tar zxvf Python-2.7.2.tgz
<span class="nb">cd </span>Python-2.7.2
./configure --prefix<span class="o">=</span>/usr/local/python2.7
make -j
make install
rm -f /usr/bin/python
<span class="c">#alternatives命令之前博客有介绍,其实就是文雅一点的替你做软连接和版本路径记录。</span>
<span class="c">#切换的时候用alternatives --config python命令选择就可以了。</span>
alternatives --install /usr/bin/python python /usr/local/python2.7/bin/python 1000
alternatives --install /usr/bin/python python /usr/bin/python2.4 500
wget http://peak.telecommunity.com/dist/ez_setup.py
python ez_setup.py
/usr/local/python2.7/bin/easy_install ElementTree
</code></pre>
</div>
<p>然后就可以试试nmap_discovery_runner.py和vmware_discovery_runner.py了。<br />
命令行方式运行如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>/opt/omd/sites/dyxmonitor/lib/shinken/libexec/nmap_discovery_runner.py -t 127.0.0.1
Got our target <span class="o">[</span><span class="s1">'127.0.0.1'</span><span class="o">]</span>
propose a tmppath /tmp/tmppTYexK
Launching <span class="nb">command</span>, sudo nmap 127.0.0.1 -T4 -O --traceroute -oX /tmp/tmppTYexK
Try to communicate
Got it <span class="o">(</span><span class="s1">'\nStarting Nmap 5.51 ( http://nmap.org ) at 2011-12-20 15:45 CST\nNmap scan report for localhost (127.0.0.1)\nHost is up (0.000030s latency).\nNot shown: 995 closed ports\nPORT STATE SERVICE\n22/tcp open ssh\n80/tcp open http\n111/tcp open rpcbind\n631/tcp open ipp\n3306/tcp open mysql\nDevice type: general purpose\nRunning: Linux 2.6.X\nOS details: Linux 2.6.15 - 2.6.31\nNetwork Distance: 0 hops\n\nOS detection performed. Please report any incorrect results at http://nmap.org/submit/ .\nNmap done: 1 IP address (1 host up) scanned in 2.05 seconds\n'</span>, <span class="s1">''</span><span class="o">)</span>
Can be <span class="o">(</span><span class="s1">'Linux'</span>, <span class="s1">'2.6.X'</span>, <span class="s1">'100'</span><span class="o">)</span>
Try to match <span class="o">(</span><span class="s1">'Linux'</span>, <span class="s1">'2.6.X'</span><span class="o">)</span>
localhost::isup<span class="o">=</span>1
localhost::os<span class="o">=</span>linux
localhost::osversion<span class="o">=</span>2.6.X
localhost::macvendor<span class="o">=</span>
localhost::openports<span class="o">=</span>22,80,111,631,3306
localhost::fqdn<span class="o">=</span>localhost
localhost::ip<span class="o">=</span>127.0.0.1
</code></pre>
</div>
<p>从输出中可以看到使用的nmap运行参数,主要是-O扫描操作系统,-T4指定快速,-oX指定输出成xml,然后用python去解析xml文件就是了。<br />
然后运行omd的命令:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>su - monitor
mkdir -p etc/shinken/objects/discovery
shinken-discovery -o /omd/sites/monitor/etc/shinken/objects/discovery -r nmap -c /omd/sites/monitor/etc/shinken/shinken-discovery.d/discovery.cfg
</code></pre>
</div>
<p>然后发现新错误:在import shinken和multiprocessing的时候有问题。<br />
因为etc/environment中需要定义PYTHONPATH=/omd/sites/monitor/lib/python:/omd/sites/monitor/lib/shinken:/usr/local/py2.7/lib/python2.7<br />
而且,因为在/omd/sites/monitor/lib/python中有multiprocessing-2.6.2.1-py2.4-linux-x86_64.egg的存在,所以导致python2.7加载py2.4的multiprocessing失败,删除掉这个目录后,让python自动加载python2.7/lib下的multiprocessing,就可以了——当然,需要先easy_install multiprocessing才有。<br />
等一会,在objects/discovery目录下给每个live的ip创建了一个文件夹,里面是host和service的cfg配置——但是因为~/bin/shinken-discovery脚本里的写法比较简单,生成的cfg里直接指定了template就是generic-host/service,然后除了description、command和host_name之外啥都木有……所以必须提前自己定义好一个模板。<br />
OMD的监控配置文件指定位置是~/etc/nagios/conf.d/,所以扫描之后,还需要整合cfg到这个目录下。</p>
<hr />
<p>最后,shinken里有bottle框架的一个webui,但是我一直没找到如何运行……shinken-specific.d/module_webui.cfg里倒是有module定义:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>define module<span class="o">{</span>
module_name WebUI
module_type webui
host 0.0.0.0 ; mean all interfaces
port 7767
auth_secret CHANGE_ME
modules Apache_passwd
<span class="o">}</span>
define module<span class="o">{</span>
module_name Apache_passwd
module_type passwd_webui
passwd /omd/sites/dyxmonitor/etc/htpasswd
<span class="o">}</span>
</code></pre>
</div>
<p>但是启动后没有看到7767端口,也没有看到任何报错——这是最让我郁闷的一点,折腾三天,全部排错都靠strace而木有log。。。<br />
所以最后还是用另一个nagios界面,trunk来启动web。trunk是一个基于perl的catalyst框架完成的页面。而omd自带了一个不小的perl5lib……这里又需要注意了:</p>
<ol>
<li>OMD安装时,会自动配置一个$PERL5LIB变量,但是不知道为啥会把<strong>/x86_64-linux-thread-multi记成</strong>/x86_64-linux,所以需要在~/etc/environment中自己重新写一次PERL5LIB。</li>
<li>因为centos5.4自带的perl版本是5.8.8;如果像我这样自己又另外安装了更高版本的perl,比如5.14.2,那么omd自带的这些perl5lib会有”undefined symbol: Perl_Gthr_key_ptr”的错误。所以老老实实用perl5.8.8好了……</li>
</ol>
<p>说老实话,觉得trunk跟classic nagios的页面没什么不同……</p>
OMD系列(二)基础配置和目录介绍
2011-12-20T00:00:00+08:00
monitor
nagios
http://chenlinux.com/2011/12/20/omd_configurations_basic
<p>话接上篇,在su和omd start之后,可以看到挂载的一个目录。其/omd是/opt/omd的软连接。目录结构如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">[</span>monitor@localhost tmp]<span class="nv">$ </span>ll /opt/omd/sites/dyxmonitor/
total 12
lrwxrwxrwx 1 monitor dyxmonitor 11 Dec 19 18:15 bin -> version/bin
drwxr-xr-x 21 monitor dyxmonitor 4096 Dec 19 18:15 etc
lrwxrwxrwx 1 monitor dyxmonitor 15 Dec 19 18:15 include -> version/include
lrwxrwxrwx 1 monitor dyxmonitor 11 Dec 19 18:15 lib -> version/lib
drwxr-xr-x 5 monitor dyxmonitor 4096 Dec 19 18:15 <span class="nb">local
</span>lrwxrwxrwx 1 monitor dyxmonitor 13 Dec 19 18:15 share -> version/share
drwxr-xr-x 14 monitor dyxmonitor 300 Dec 19 23:32 tmp
drwxr-xr-x 12 monitor dyxmonitor 4096 Dec 19 18:15 var
lrwxrwxrwx 1 monitor dyxmonitor 19 Dec 19 18:15 version -> ../../versions/0.50
</code></pre>
</div>
<p>软连接的部分,都是所有omd共用的;tmp就是挂载的tmpfs用来加速数据读写的;var是日志和由前端生成的数据(比如rrd)存放地点;etc是配置文件存放地点,具体内容包括:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">[</span>monitor@localhost etc]<span class="nv">$ </span>ls
apache cron.d htpasswd logrotate.conf mod-gearman nsca rc.d thruk
check_mk dokuwiki init.d logrotate.d nagios omd rrdcached.conf xinetd.conf
check_multi environment jmx4perl mk-livestatus nagvis pnp4nagios shinken xinetd.d
</code></pre>
</div>
<p>其中,htpasswd定义了auth的用户名密码,默认是omdadmin:omd;<br />
rrdcached.conf定义了rrdcached的过期时间,用来缓解rrdtools的压力;<br />
apache/mode.conf是/opt/omd/apache/monitor.conf里Include的文件,实质是apache/apache-own.conf的软连接;调用了apache/proxy-port.conf来反向代理5000端口的实质页面。<br />
apache/apache.conf是真正的5000端口运行的配置文件,里面Include了apache/conf.d/*.conf——这些conf都是外面不同插件目录里的apache.conf的软连接:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">[</span>monitor@localhost conf.d]<span class="nv">$ </span>ll
total 24
-rw-r--r-- 1 monitor monitor 119 Dec 19 18:15 01_python.conf
-rw-r--r-- 1 monitor monitor 482 Dec 19 18:15 02_fcgid.conf
-rw-r--r-- 1 monitor monitor 252 Dec 19 18:15 auth.conf
lrwxrwxrwx 1 monitor monitor 26 Dec 19 18:15 check_mk.conf -> ../../check_mk/apache.conf
lrwxrwxrwx 1 monitor monitor 26 Dec 19 18:15 dokuwiki.conf -> ../../dokuwiki/apache.conf
lrwxrwxrwx 1 monitor monitor 25 Dec 19 23:29 nagios.conf -> ../../shinken/apache.conf
lrwxrwxrwx 1 monitor monitor 24 Dec 19 18:15 nagvis.conf -> ../../nagvis/apache.conf
-rw-r--r-- 1 monitor monitor 519 Dec 19 18:15 omd.conf
lrwxrwxrwx 1 monitor monitor 28 Dec 19 18:15 pnp4nagios.conf -> ../../pnp4nagios/apache.conf
-rw-r--r-- 1 monitor monitor 117 Dec 19 18:15 site.conf
lrwxrwxrwx 1 monitor monitor 23 Dec 19 18:15 thruk.conf -> ../../thruk/apache.conf
-rw-r--r-- 1 monitor monitor 283 Dec 19 18:15 var_www.conf
</code></pre>
</div>
<p>另外,在etc下还有omd/site.conf配置文件。这是本site的主配置文件,看起来可能不太清楚,所以omd提供了更直观的修改办法,那就是仿UI的omd config命令:<br />
<img src="/images/uploads/omd.png" alt="" title="omd" width="300" height="173" class="alignnone size-medium wp-image-2830" /><br />
通过这种类似setup的方式直接搞定就可以了。<br />
默认情况下,web页面的首页访问url地址是/monitor/omd/,这个页面上列出了classic nagios、check_mk、nagvis、pnp4nagios和dokuwiki的访问效果截图和地址,可以点击进入查看。然后再用omd config定义Web UI的default选择就是了。一旦定义完成,配置会自动修改,下次再访问/monitor/omd/,就会自动跳转了。<br />
注:/monitor/是因为我create的site名字是monitor<br />
另,修改config后,发现mod_gearman不可用。原因是omd的rpm发布里没有带gearmand的实现,必须自己另外搞定gearman的jobserver~~</p>
OMD系列(一)简介与安装
2011-12-19T00:00:00+08:00
monitor
nagios
http://chenlinux.com/2011/12/19/omd_intro_install_on_centos5
<p>OMD,全称Open Monitoring Distribution,是一个围绕Nagios core构建的分布式开源监控集。在nagios基础上融合了NRPE、NSCA、check_mk、mod_gearman、pnp4nagios、nagvis、rrdcached等插件,以完成高性能的、可视化的,分权限管理的监控系统。<br />
(我是在看mod_gearman的安装介绍时看到的,感觉这种一体式的安装很爽)<br />
项目主页是<a href="http://omdistro.org" target="_blank">http://omdistro.org</a>,提供了rh、debian、suse和src各种安装模式。<br />
比如在centos5上面,只需要简单的操作即可:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm
<span class="c">#单独装graphviz是因为epel里的版本有冲突</span>
yum install graphviz-gd.x86_64
<span class="c">#已经有rhel6上的版本,而5.4提供最高只到0.45,mod_gearman从0.48才加入,所以试试5.5的,发现也没问题~~</span>
yum install --nogpgcheck http://omdistro.org/attachments/download/121/omd-0.50-rh55-25.x86_64.rpm
</code></pre>
</div>
<p>然后运行omd create monitor即可。<br />
omd会自动在linux系统上添加一个monitor用户,然后其他操作可以在su - monitor后再继续,这样比较安全,而且可以看到的是,create的时候,还挂载了tmpfs到/omd/sites/monitor/tmp,以提供更高的性能。<br />
切换到monitor用户后,运行omd start,即可启动。<br />
先写到这里,慢慢看具体配置。</p>
Chart::OFC试用
2011-12-19T00:00:00+08:00
perl
http://chenlinux.com/2011/12/19/intro_open_flash_chart
<p>这几天ppt看的比较多,一样一样来玩。今天先说OFC,之前有试过amcharts和fusioncharts。amcharts最全最漂亮(尤其是有scroll和map),可惜图上都自动标上了amcharts.com的字样;fusioncharts的free版本,也蛮好用的,虽然scroll怪怪的居然靠的是点击控制。比较相同的一点就是,这两都只提供了php、python等的模块,如果是perl写的网页,那只能通过提供xml/json数据给js控制的办法。稍微一点点遗憾……<br />
这次在cpan上终于看到一个chart控制的模块,叫Chart::OFC。这个OFC的项目地址如下:<br />
<a href="http://teethgrinder.co.uk/open-flash-chart/" target="_blank">http://teethgrinder.co.uk/open-flash-chart/</a><br />
其实用法上没什么特殊,无非是省略一点点xml代码,通过OO的方式自动生成而已。让我觉得蛮好玩的是官网上作者的声明。因为他曾经在维护公司一个付费的flash chart项目时,给甲方发信要求修改bug,等了一个月没反应。于是自己现学as语言开始自己搞= =!然后念念不忘的提示说:要重视客户的反馈。。。。。。哈哈<br />
这个项目目前用as3改写,所以新版本叫OFC2了,不过作者自己也说不太稳定,建议继续用OFC1.9.7,所以先不说Chart::OFC2,继续用Chart::OFC好了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>
<span class="nv">get</span> <span class="s">'/ofc_data'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="k">use</span> <span class="nn">Chart::</span><span class="nv">OFC</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$line_array</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span> <span class="o">..</span> <span class="mi">20</span><span class="p">];</span>
<span class="k">my</span> <span class="nv">$bars_array</span><span class="p">;</span> <span class="nv">@$bars_array</span> <span class="o">=</span> <span class="nb">map</span> <span class="p">{</span><span class="nv">$_</span> <span class="o">-</span> <span class="mf">2.5</span><span class="p">}</span> <span class="nv">@$line_array</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$leng</span> <span class="o">=</span> <span class="nv">$#$line_array</span> <span class="o">+</span> <span class="mi">1</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$bars</span> <span class="o">=</span> <span class="nn">Chart::OFC::Dataset::</span><span class="nv">Bar</span><span class="o">-></span><span class="k">new</span><span class="p">(</span> <span class="nb">values</span> <span class="o">=></span> <span class="nv">$bars_array</span> <span class="p">);</span>
<span class="k">my</span> <span class="nv">$line</span> <span class="o">=</span> <span class="nn">Chart::OFC::Dataset::</span><span class="nv">Line</span><span class="o">-></span><span class="k">new</span><span class="p">(</span> <span class="nb">values</span> <span class="o">=></span> <span class="nv">$line_array</span> <span class="p">);</span>
<span class="k">my</span> <span class="nv">$x_axis</span> <span class="o">=</span> <span class="nn">Chart::OFC::</span><span class="nv">XAxis</span><span class="o">-></span><span class="k">new</span><span class="p">(</span> <span class="nv">axis_label</span> <span class="o">=></span> <span class="s">'X Axis'</span><span class="p">,</span> <span class="nv">label_steps</span> <span class="o">=></span> <span class="mi">2</span><span class="p">,</span> <span class="nv">tick_steps</span> <span class="o">=></span> <span class="mi">2</span><span class="p">,</span> <span class="nv">labels</span> <span class="o">=></span> <span class="nv">$line_array</span> <span class="p">);</span>
<span class="k">my</span> <span class="nv">$y_axis</span> <span class="o">=</span> <span class="nn">Chart::OFC::</span><span class="nv">YAxis</span><span class="o">-></span><span class="k">new</span><span class="p">(</span> <span class="nv">axis_label</span> <span class="o">=></span> <span class="s">'Y Axis'</span><span class="p">,</span> <span class="nv">max</span> <span class="o">=></span> <span class="nv">$leng</span><span class="p">,</span> <span class="nv">label_steps</span> <span class="o">=></span> <span class="mi">2</span> <span class="p">);</span>
<span class="k">my</span> <span class="nv">$grid</span> <span class="o">=</span> <span class="nn">Chart::OFC::</span><span class="nv">Grid</span><span class="o">-></span><span class="k">new</span><span class="p">(</span> <span class="nv">title</span> <span class="o">=></span> <span class="s">'My Grid Chart'</span><span class="p">,</span>
<span class="nv">datasets</span> <span class="o">=></span> <span class="p">[</span> <span class="nv">$bars</span><span class="p">,</span> <span class="nv">$line</span> <span class="p">],</span>
<span class="nv">x_axis</span> <span class="o">=></span> <span class="nv">$x_axis</span><span class="p">,</span>
<span class="nv">y_axis</span> <span class="o">=></span> <span class="nv">$y_axis</span><span class="p">,</span>
<span class="p">);</span>
<span class="k">return</span> <span class="nv">$grid</span><span class="o">-></span><span class="nv">as_ofc_data</span><span class="p">();</span>
<span class="p">};</span>
<span class="nv">get</span> <span class="s">'/ofc_test'</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="nv">template</span> <span class="s">'ofc'</span><span class="p">;</span>
<span class="p">};</span>
</code></pre>
</div>
<p>然后ofc.tt是这样:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>
<span class="nt"><html><head></span>
<span class="nt"><meta</span> <span class="na">http-equiv=</span><span class="s">"content-type"</span> <span class="na">content=</span><span class="s">"text/html; charset=UTF-8"</span><span class="nt">></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><script </span><span class="na">type=</span><span class="s">"text/javascript"</span> <span class="na">src=</span><span class="s">"/ofc/js/swfobject.js"</span><span class="nt">></script></span>
<span class="nt"><div</span> <span class="na">id=</span><span class="s">"my_chart"</span><span class="nt">></div></span>
<span class="nt"><script </span><span class="na">type=</span><span class="s">"text/javascript"</span><span class="nt">></span>
<span class="kd">var</span> <span class="nx">so</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">SWFObject</span><span class="p">(</span><span class="s2">"/ofc/actionscript/open-flash-chart.swf"</span><span class="p">,</span> <span class="s2">"ofc"</span><span class="p">,</span> <span class="s2">"500"</span><span class="p">,</span> <span class="s2">"200"</span><span class="p">,</span> <span class="s2">"9"</span><span class="p">,</span> <span class="s2">"#FFFFFF"</span><span class="p">);</span>
<span class="nx">so</span><span class="p">.</span><span class="nx">addVariable</span><span class="p">(</span><span class="s2">"data"</span><span class="p">,</span> <span class="s2">"/ofc_data"</span><span class="p">);</span>
<span class="nx">so</span><span class="p">.</span><span class="nx">addParam</span><span class="p">(</span><span class="s2">"allowScriptAccess"</span><span class="p">,</span> <span class="s2">"always"</span> <span class="p">);</span><span class="c1">//"sameDomain");</span>
<span class="nx">so</span><span class="p">.</span><span class="nx">write</span><span class="p">(</span><span class="s2">"my_chart"</span><span class="p">);</span>
<span class="nt"></script></span>
<span class="nt"><div></span>boo<span class="nt"></div></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</code></pre>
</div>
<p>运行即可。<br />
这里chart主要分两种,pie和grid。<br />
bar和line都是grid的。Chart::OFC的pod里举例是pie的,我这里写的举例是grid的,来自Chart::OFC::Grid的说明;<br />
在DataSet中可以看到,area、candle和scatter都是从Line.pm模块extends出来的(嗯,这个模块是用Moose构建滴),所以具体生成的时候都是用grid。<br />
<img src="/images/uploads/ofc.png" alt="" title="ofc" width="600" height="240" class="alignnone size-full wp-image-2824" /></p>
<hr />
<p>另外,CPAN上还有一个模块是给XML/SWF Chart生成数据的,不过那个东东也是要buy滴……<br />
再另,有木有仪表盘的charts可用呢……</p>
Dancer::Plugin::SimpleCRUD模块学习
2011-12-15T00:00:00+08:00
dancer
perl
MySQL
http://chenlinux.com/2011/12/15/learning_dancer_plugin_simplecrud
<p>Dancer圣临历中介绍了一个新插件Dancer::Plugin::SimpleCRUD。可以很快的生成对数据库表的create/read/update/delete。大概阅读了一下代码。主要就是使用HTML::FromDatabase模块生成read的页面,用CGI::FormBuilder模块生成create/update/delete的页面,用Dancer::Plugin::Database::Handle模块操作数据库。因为是Simple,所以html代码都是固定的,不甚美观。但是思路可以学习,通过这两个模块,可以自己结合Dancer的template系统做CRUD了。<br />
先来说HTML::FromDatabase模块,这个只涉及select,所以特别简单:<br />
<code class="highlighter-rouge">perl
any ['get', 'post'] => '/add/:element' => sub {
my $element = params->{'element'};
if ( request->method eq 'GET' ) {
my $sth = database->prepare("select * from $element");
$sth->execute();
my $table = HTML::Table::FromDatabase->new( -sth => $sth )->getTable;
template 'info', { table => $table };
};
...
};
</code></p>
<p>这样就可以了,getTable方法的返回是一个字符串(内容就是table/tr/td代码),直接传递给tmpl就行了~~<br />
然后看CGI::FromBuilder模块:<br />
```perl<br />
any [‘get’, ‘post’] => ‘/add/:element’ => sub {<br />
my $element = params->{‘element’};<br />
my $paramsobj = request->method eq ‘POST’ ? { params => { params() } } : undef;<br />
my @fields;<br />
if ( $element eq ‘food’ ) {<br />
@fields = qw/fid name shelf/;<br />
} elsif ( $element eq ‘market’ ) {<br />
@fields = qw/mid name location/;<br />
} elsif $element eq ‘price’ ) {<br />
@fields = qw/fid mid price time/;<br />
} else {<br />
return “Error element”;<br />
};<br />
my $form = CGI::FormBuilder->new(<br />
params => $paramsobj,<br />
fields => \@fields,<br />
);</p>
<div class="highlighter-rouge"><pre class="highlight"><code>my $sth = database->prepare("select * from $element order by fid desc");
$sth->execute;
my $default_hashref = $sth->fetchrow_hashref;
my $action = $default_hashref ? 'update' : 'insert';
if ( request->method eq 'POST' && $form->submitted && $form->validate ) {
my $success = database->quick_update("$element", {name => params->{'name'}}, {shelf => params->{'shelf'} });
redirect "/add/$element" if $success;
return "update error";
} else { # return $form->render(values => $default_hashref,
my $hash = $form->prepare(values => $default_hashref,
title => $element,
action => "/add/$element",
method => "post",
);
template 'crud', { hash => $hash, };
}; }; ```
</code></pre>
</div>
<p>这里的关键,就是$paramsobj变量。否则无法区分GET和POST的。<br />
和HTML::FromDatabase一样,有个render()直接返回内容是组装好的html代码的字符串;另外,CGI::FormBuilder模块还提供一个prepare()方法,返回的是hash——因为render()返回的html代码是完整的html/head/body…(事实上CGI::FormBuilder模块的构造器还提供指定js函数和css格式的参数);所以如果要加进template里,可以把prepare()返回的hash传递给tmpl去操作!<br />
然后还发现一个小问题,如果table里一条数据都没有,fields为undef,模块运行有问题的……<br />
最后,<em>quick_update()是Dancer::Plugin::Database::Handle模块里提供的,这个模块里提供了一系列_quick</em>*()方法,用来快速操作数据库。</p>
nginx_perl试用
2011-12-12T00:00:00+08:00
nginx
perl
nginx
http://chenlinux.com/2011/12/12/nginx_perl_demo
<p>因为空闲时间比较多,所以在CPAN上乱翻,看到了nginx_perl这个项目(原名Nginx::Engine),现在托管在github.com上。地址见:<br />
<a href="https://github.com/zzzcpan/nginx-perl">https://github.com/zzzcpan/nginx-perl</a></p>
<p>这个模块的目的,是在nginx内置官方perl模块的基础上,实现一系列异步非阻塞的api。用connector/writer/reader完成类似proxy的功能(这里因为是交给perl完成,所以不单单局限在http上了),用take_connection/give_connection完成类似websocket的功能,用ssl_handshaker完成ssl的解析,用timer完成定时器,用resolver完成域名解析…使用方法简单来说,就是用main_count_inc/finalize_request控制计数器,用NGX_READ/NGX_WRITE/NGX_CLOSE等控制callback。</p>
<p>其他内容和apache的mod_perl,或者nginx.org的perl类似。最新版的POD地址见:<a href="http://zzzcpan.github.com/nginx-perl/Nginx.html">http://zzzcpan.github.com/nginx-perl/Nginx.html</a></p>
<p>最后举例一个自己写的简单的例子:</p>
<p><code class="highlighter-rouge">perl
package HelloWorld;
use Nginx;
use strict;
#用来在nginx启动的时候做的事情,这里单纯显示一下
sub init_worker {
warn 'nginx_perl start [OK]';
};
#这里是nginx的http模块中调用的handler,alexander有计划改成tcp级别的
sub handler {
my $r = shift;
#增加主循环的计数器
$r->main_count_inc;
#使用非阻塞的连接器连接127.0.0.1的80端口,10秒超时
ngx_connector '127.0.0.1', 80, 10, sub {
#如果连接出问题,会记录在$!中
if ($!) {
$r->send_http_header('text/html');
$r->print("Connection failed, $!\n");
$r->send_special(NGX_HTTP_LAST);
#不管怎么处理这次连接,最后一定要记得用$r->finalize_request(),会decrease之前$r->main_count_inc;里加上的计数。
$r->finalize_request(NGX_OK);
return NGX_CLOSE;
};
#返回$c是建立的连接
my $c = shift;
my $req_buf = "GET /index.php HTTP/1.0\x0d\x0a".
"Host: chenlinux.com\x0d\x0a".
"Connection: close\x0d\x0a".
"\x0d\x0a";
#这里记住定义buffer的时候不要搞成undef了,会报段错误的,不过俄国佬回信说他修复了
my $res_buf = '';
#非阻塞写入,超时10秒
ngx_writer $c, $req_buf, 10, sub {
if ($!) {
......
};
$req_buf = '';
#之前的buffer测试就是这里,如果加一个warn,就不会报错……汗
#warn "$req_buf\n$res_buf\n";
#写入完成后,开始调用读取
return NGX_READ;
};
#读取到buffer,最短0字节,最长8000字节,超时10秒
ngx_reader $c, $res_buf, 0, 8000, 10, sub {
if ($!) {
......
}
$r->send_http_header('text/html');
$r->print($res_buf);
$r->send_special(NGX_HTTP_LAST);
$r->finalize_request(NGX_OK);
return NGX_CLOSE;
};
#这个是connector的语句,表示连接成功后调用写入
return NGX_WRITE;
};
#各个非阻塞调用完成后的返回,NGX_DONE只能用在http的handler里,不能在ngx_*r里用,里面请用NGX_CLOSE。
return NGX_DONE;
};
1;</code></p>
<p>另:源码中带有一个真正的反向http的例子Nginx::Util和一个Redis的例子。并且与nodejs读取redis的性能做了对比。可以参见~~</p>
<p><strong style="display:block;margin:12px 0 4px"><a href="http://www.slideshare.net/chenryn/perlnginx" title="Perl在nginx里的应用" target="_blank">Perl在nginx里的应用</a></strong></p>
websocket体验
2011-11-04T00:00:00+08:00
dancer
websocket
http://chenlinux.com/2011/11/04/test_websocket_with_dancer
<p>因为看到mojo在宣传其内置websocket支持,去CPAN上search了一下,发现Dancer也有plugin做websocket用。稍微看了一下,很显然没看mojo的内嵌代码,而dancer的plugin代码倒是相当简单。<br />
简单的说,就是通过AnyMQ、Plack和Web::Hippie的组合完成。注意的是,因为Web::Hippie使用了AnyEvent::Handle管理websocket的fh,AnyMQ也使用AnyEvent管理message queue,所以在启动plackup的时候,必须使用AnyEvent核心的Twiggy服务器。<br />
Plugin里硬编码很多。比较重要的地方就是”set plack_middlewares_map”,plack_middlewares_map是dancer新加的功能,这里用URLMap来mount ‘/_hippie’到Web::Hippie::Pipe和AnyMQ上。也就是说,使用websocket的访问路径,必须固定为’^/_hippie/.*‘这样。<br />
然后像两个get方法,’/new_listener’和’/message’,这两个访问路径是Web::Hippie里写死的,不能改。好在一般应用也不会直接访问这个。<br />
最后有三个register到Dancer的方法,’ws_on_message’、’ws_on_new_listener’和’ws_send’,前两个用来覆盖上面提到的get的两个route的具体信息,一般不用前面两个,因为这样就意味着页面的js里无法控制websocket了(个人理解,不知道对否?)后面那个类似把websocket改成普通的params方式请求响应(用curl/wget可以请求的那种)。</p>
<hr />
<p>好了,大概的机理就是这样,现在做个小东西试试看。cpanm和dancer的运用之前写过,就略过了。lib/WS_test.pm如下:<br />
<code class="highlighter-rouge">perlpackage WS_test;
use Dancer ':syntax';
use Dancer::Plugin::WebSocket;
get '/ws' => sub {
template 'websocket';
};
true;</code><br />
views/websocket.tt如下:<br />
```perl</p>
<html>
<head>
<script>
var ws_path = "ws://chenlinux.com:8080/_hippie/ws";
var socket = new WebSocket(ws_path);
socket.onopen = function() {
document.getElementById('conn-status').innerHTML = 'Connected';
};
socket.onmessage = function(e) {
var data = JSON.parse(e.data);
if (data.msg)
document.getElementById('textmsg').value += data.name + data.msg + "\n";
};
function send_msg() {
var name = document.getElementById('name').value + ": ";
var talk = document.getElementById('ownmsg').value;
socket.send(JSON.stringify({ name: name,msg: talk }));
}
</script>
</head>
<body>
Connection Status: <br /><span id="conn-status"> Disconnected </span><br />
<textarea id="textmsg" cols="55" rows="10"></textarea><hr />
nickname: <input type="text" id="name" /><br />
messages: <textarea id="ownmsg"></textarea><br />
<input value="send" type=button onclick="send_msg()" />
</body>
</html>
<p><code class="highlighter-rouge">
ok,然后启动server就行了:
</code>bashsudo twiggy –listen :8080 bin/app.pl```<br />
然后访问http://chenlinux.com:8080/ws/看看效果吧~一个小聊天室就出来了。<br />
不过还有点问题,就是根据加入聊天室的时间先后,信息看不全……估计应该是AnyMQ的问题,个人对MQ了解不多,还需要继续看代码……</p>
用perl调用新浪微博API小实验
2011-11-04T00:00:00+08:00
dancer
perl
oauth
http://chenlinux.com/2011/11/04/perl_oauth_to_weibo_api
<p>新浪微博API是通过OAuth方式的。perl上有现成的模块,只需要稍微调一下参数就行了。下午比较闲,试试看。。。<br />
一般网上说的,都还是Net::OAuth模块,这个是1.0a版本的,一看perldoc那个长篇就头疼。这里我用Net::OAuth2模块,简单方便,而且正好配合dancer框架,方便做外部网站应用。<br />
首先是安装模块:<br />
<code class="highlighter-rouge">perlcpanm --mirror-only --mirror http://mirrors.163.com/cpan LWP::Protocol::https Net::OAuth2::Client</code><br />
因为新浪微博的api用的是https协议,所以要加上LWP::Protocol::https模块,这样才能发起443请求。<br />
然后在web/lib/weibo.pm里里添加如下语句:<br />
```perluse Net::OAuth2::Client;<br />
use JSON qw(decode_json); #只需要decode_json,因为dancer自己有from_json和to_json,不要覆盖了</p>
<p>get ‘/oauth’ => sub { #这个url是对外发布的地址,用来跳转到微博开放平台验证是否授权。<br />
my $client = &test_client;<br />
redirect $client->authorize_url;<br />
};</p>
<p>get ‘/oauth/callback’ => sub { #这个url是授权返回地址,真正的应用入口。<br />
my $client = &test_client;<br />
#这个code参数是前面的authorize_url授权验证后返回时带上的,使用这个code进行access token验证。<br />
#所以无法直接访问/oauth/callback,必须通过/oauth访问。<br />
my $access_token = $client->get_access_token(params->{code});<br />
#access token验证通过后,真正的对api发起请求,列表见http://open.weibo.com/wiki/API文档_V2<br />
#请求方法有get/post/delete等,见Net::OAuth2::AccessToken模块。<br />
my $response = $access_token->get(‘/2/statuses/user_timeline.json’);<br />
if ( $response->is_success ) {<br />
my $data = decode_json $response->decoded_content;<br />
template ‘weibo’, { msgs => $data };<br />
# return $data->{‘statuses’}->[0]->{‘text’};<br />
} else {<br />
return $response->status_line;<br />
};<br />
};</p>
<p>sub test_client {<br />
Net::OAuth2::Client->new(<br />
config->{app_key},<br />
config->{app_secret},<br />
user_agent => LWP::UserAgent->new(ssl_opts => {SSL_verify_mode => ‘0x01’}),<br />
site => ‘https://api.weibo.com’,<br />
authorize_path => ‘/oauth2/authorize’,<br />
access_token_path => ‘/oauth2/access_token’,<br />
access_token_method => ‘POST’,<br />
)->web_server(<br />
redirect_uri => uri_for(‘/oauth/callback’) #注意这个url需要去新浪授权,否则验证cb地址不匹配会报错的. <br />
);<br />
};```<br />
然后在config.yml里定义好从新浪微博开放平台里申请应用给的key和secret就行了。</p>
<p>这里跟模块里提供的demo主要有几点不一样的地方:</p>
<p>第一个是test_client必须赋值给变量,不知道为啥demo里不用?<br />
第二个是Net::OAuth2::Client->new里,需要自己创建useragent,默认的useragent只是LWP::UserAgent->new,没有ssl_opts,这样会在callback的时候反馈ssl验证有问题。<br />
第三个是access_token_method,虽然新浪的开发文档上写的GET,但实际只能用POST。</p>
<p>最后,测试成功返回了json数据,<del datetime="2011-11-05T07:33:45+00:00">但是还有两个问题:
第一,直接return $data->{‘statuses’}->[0]->{‘text’};看到的是乱码,应该是content编码设置问题;</del><br />
第二,使用template如下:<br />
```perl<% FOREACH status IN msgs.statuses %><br />
when: <% status.created_at %><br /><br />
text: <% status.text %><br /><br />
from: <% status.source %><br /><br />
user: <% status.user.name %><br /><br />
city: <% status.user.location %><br /></p>
<hr />
<p><% END %>```<br />
<del datetime="2011-11-04T11:01:17+00:00">但是访问页面http://chenlinux.com:8080/oauth后跳转到页面显示的居然还是HASH(0X1234)这样的地址。奇怪了。先记录一下,有空继续调……</del><br />
<strong>补充:第二个问题解决了,原因居然是config.yml里忘了用TT模版,simple模板不会for循环的……
继续补充:第一个问题也解决了,原因是不能直接用from_json,而要用decode_json。</strong><br />
效果如下:<br />
<img src="/images/uploads/weibo_api.png" alt="" title="weibo_api" width="601" height="370" class="alignnone size-full wp-image-2683" /></p>
一个perl扩展正则表达式
2011-10-26T00:00:00+08:00
perl
http://chenlinux.com/2011/10/26/a_perl_extended_regex
<p>今天傍晚,莫言在Q群里贴了一个他写的正则表达式,回来翻了翻perlre文档,基本算是看懂,赶紧记录下来:<br />
<code class="highlighter-rouge">perlmy $ip = "192.168.0.1|192.168.0.2|192.168.0.1";
if ( $ip =~ /
^
(?:
((?:\d{1,3}\.){3}\d{1,3})
(?=
(?:
\|(?!\1)(?1)
)*
\z
)
\|
)*
(?1)
$
/x ) {
print "match\n";
}</code><br />
根据<a href="http://perldoc.perl.org/perlre.html" title="perlre文档" target="_blank">perlre文档</a>的说明,一点一点解释。</p>
<ol>
<li>首先是/x,用这个来去除regex里的空格,不然的话写在一行太难看懂了;</li>
<li>然后是^,表示从最开头开始;</li>
<li>然后是(?:,这个表示本括号不记入反向引用$&中;</li>
<li>然后是((?:\d{1,3}.){3}\d{1,3}),同样里面一个(?:,也就是说这一行匹配一个ip,并计为$1;</li>
<li>然后是(?=,这个表示在上面那行ip的正则后面必须出现符合本括号定义,同样也不计入$&(术语叫”零宽肯定前向断言”是吧?);</li>
<li>
<table>
<tbody>
<tr>
<td>然后一个隔开ip的</td>
<td>;</td>
</tr>
</tbody>
</table>
</li>
<li>然后是(?!,这个表示本括号内的东西绝对不能出现,同样也不计入$&(术语叫”零宽否定前向断言”是吧?);</li>
<li>
<table>
<tbody>
<tr>
<td>然后是\1,这个就是前面捕获的$1,跟上行解释的断言合在一起,就是</td>
<td>后面不能有和前面匹配的ip重复;</td>
</tr>
</tbody>
</table>
</li>
<li>然后是(?1,这个表示前面捕获$1的正则表达式,也就是不重复ip的情况下,继续捕获新ip;</li>
<li>
<table>
<tbody>
<tr>
<td>然后是)*,这个)闭合到</td>
<td>前面的(?:,也就是说</td>
<td>ip可以重复多个;</td>
</tr>
</tbody>
</table>
</li>
<li>然后是\z,这个是字符串边界,相当于单行里$的作用,在本例中可以互换,用在这里,就是为了让(?!\1)的检查一直执行到最后;</li>
<li>然后是),闭合(?=;</li>
<li>
<table>
<tbody>
<tr>
<td>然后是|和)*,这里闭合到^(,表示符合不重复ip条件的ip</td>
<td>格式不断正则匹配;</td>
</tr>
</tbody>
</table>
</li>
<li>然后是(?1)$,定义最后一个ip,使用和$1相同的正则,也就是字符串至少要有一个ip。</li>
</ol>
<p>OK,解释完毕。其实,从后往前看,反而清晰一些~~</p>
<p>另:perlre中在(??{CODE})段的表述中有如下一段话“In perl 5.12.x and earlier, because the regex engine was not re-entrant, delayed code could not safely invoke the regex engine either directly with “m//” or “s///”), or indirectly with functions such as “split”.”,而(?R)和(??{CODE})做的是类似而简单的任务,所以如果linux发行版里带的perl版本不够高的话,这里就不能用(?1)的简单写法,需要自己再写一遍了。</p>
<p>可以这么判断:<br />
<code class="highlighter-rouge">perl
my $re = $^V lt v5.14 ? '(?:\d{1,3}\.?){4}' : '(?1)';
my $ip = "192.168.0.1|192.168.0.2|192.168.0.3|192.168.0.4|192.168.0.5";
if ( $ip =~ m/
^
(?:
((?:\d{1,3}\.?){4})
(?=
(?:
\|(?!\1)$re
)*
\z
)
\|
)*
$re
$
/x ) {
print "$1 match\n";
}
</code></p>
用Template::Tookit给squid.conf写模板
2011-10-25T00:00:00+08:00
perl
http://chenlinux.com/2011/10/25/rewrite_template_for_squid-conf
<p>本文纯属练习Template模块使用,是否可以运用到生产,是否有必要运用到生产,都是未知数……<br />
包括如下文件:<br />
```bash[raocl@localhost tt2-test]$ tree<br />
.<br />
|– config-cdcgame.net.yml<br />
|– config-china.com.yml<br />
|– config.tt<br />
|– hostconfig.yml<br />
|– squid.layout.tt<br />
`– tt4squid.pl</p>
<p>0 directories, 6 files<code class="highlighter-rouge">
其中tt4squid.pl如下:
</code>perl#!/usr/bin/perl<br />
use warnings;<br />
use strict;<br />
use Template;<br />
use YAML::Syck;</p>
<p>my $config_path = ‘./’;<br />
my $data = LoadFile(“${config_path}hostconfig.yml”);<br />
$data->{‘configs’} = \&loadconfigs;<br />
my $tt = Template->new;<br />
$tt->process(“$ARGV[0]”, $data) or die $tt->error;</p>
<p>sub loadconfigs {<br />
my @ref_array;<br />
my @ymls = grep {s/${config_path}config-(.+?.yml)/$1/} glob(“${config_path}<em>”);<br />
foreach my $yml (@ymls) {<br />
my $hash_ref = LoadFile(“${config_path}config-${yml}”);<br />
push @ref_array, $hash_ref;<br />
};<br />
return \@ref_array;<br />
};<code class="highlighter-rouge">
config.tt模板如下:
</code>perl[%# 用%后面紧跟的#表示注释。用%紧跟的-表示消除外面的一个\s。 %]<br />
[%# 用WRAPPER表示加入layout模板,这个跟INCLUDE/PROCESS有点不同,之前Dancer的时候用过 %]<br />
[% WRAPPER squid.layout.tt -%]<br />
[% FOREACH config IN configs %]<br />
####[% config.custom %]<br />
[% IF config.rewrite -%]<br />
acl [% config.custom %]_url_rewrite url_regex -i [% config.rewrite.url_regex %]<br />
url_rewrite_access deny ![% config.custom %]_url_rewrite<br />
url_rewrite_program [% config.rewrite.program %]<br />
url_rewrite_concurrency [% config.rewrite.concurrency %]<br />
[% END -%]<br />
[% IF config.cache_deny_list -%]<br />
[% FOREACH list IN config.cache_deny_list -%]<br />
acl no_cache_acl4[% config.custom %] url_regex -i [% list %]<br />
[% END -%]<br />
cache deny no_cache_acl4[% config.custom %]<br />
[% END -%]<br />
[% IF config.http_access_list -%]<br />
[% FOREACH prior_list IN config.http_access_list -%]<br />
[% FOREACH list IN prior_list -%]<br />
acl acl_[% config.custom %]_[% list.access %]_[% list.priority %] url_regex -i [% list.url_regex %]<br />
[% END -%]<br />
[%# 这里虽然END退出了循环,但是原来内存里的数据没有清除,所以下一行的list数据结构就是上面循环的最后一次执行结果 %]<br />
http_access [% list.access %] acl_[% config.custom %]_[% list.access %]_[% list.priority %]<br />
[% IF list.allow_referer -%]<br />
acl not_null_referer referer_regex -i .<br />
acl [% config.custom %]_allow_referer referer_regex -i<br />
[%- FOREACH referer IN list.allow_referer -%]<br />
[% referer -%]<br />
[% END %]<br />
http_access allow acl_[% config.custom %]_[% list.access %]_[% list.priority %] !not_null_referer<br />
http_access deny acl_[% config.custom %]_[% list.access %]_[% list.priority %] [% config.custom %]_allow_referer<br />
[% END -%]<br />
[% IF config.deny_info -%]<br />
deny_info [% config.deny_info %] acl_[% config.custom %]_[% list.access %]_[% list.priority %]<br />
[% END -%]<br />
[% END -%]<br />
[% END -%]<br />
[% IF config.refresh_patterns -%]<br />
[% FOREACH pattern IN config.refresh_patterns -%]<br />
refresh_pattern -i [% pattern.url_regex %] [% pattern.min %] [% pattern.per %]% [% pattern.max %]<br />
[%- FOREACH option IN pattern.options -%]<br />
[% option -%]<br />
[% END -%]<br />
[% END -%]<br />
[% END -%]<br />
[% END %]<br />
[% END %]<code class="highlighter-rouge">
通过WRAPPER加载的squid.layout.tt模板如下:
</code>squid#################ACL1############################<br />
acl all src 0.0.0.0/0.0.0.0<br />
#############################################<br />
http_port [% http_port %] accel vhost vport http11 allow-direct<br />
icp_port 0<br />
acl shoutcast rep_header X-HTTP09-First-Line ^ICY.[0-9]<br />
upgrade_http0.9 deny shoutcast<br />
negative_ttl [% negative_ttl %] second<br />
refresh_stale_hit 0 minute<br />
vary_ignore_expire on<br />
acl apache rep_header Server ^Apache<br />
broken_vary_encoding allow apache<br />
cache_vary on<br />
cache_mgr [% admin_email %]<br />
visible_hostname [% local_hostname %]<br />
icp_access deny all<br />
cache_effective_user nobody<br />
cache_effective_group nobody<br />
httpd_suppress_version_string on<br />
debug_options ALL,1<br />
#####################################<br />
pipeline_prefetch on<br />
pid_filename /var/run/squid.pid<br />
hierarchy_stoplist<br />
[%- FOREACH stop IN stoplist -%]<br />
[% stop -%]<br />
[% END %]<br />
######################################<br />
cache_mem [% cache_mem %] MB<br />
maximum_object_size_in_memory [% max_in_mem %] KB<br />
maximum_object_size [% max_obj %] MB<br />
minimum_object_size 0 KB<br />
[% FOREACH coss IN cossdirs -%]<br />
cache_dir coss [% coss.dir %] [% coss.dir_size %] max-size=[% coss.max_size %] block-size=[% coss.block_size %] membufs=[% coss.membufs %]<br />
[% END -%]<br />
[% FOREACH aufs IN aufsdirs -%]<br />
cache_dir aufs [% aufs.dir %] [% aufs.dir_size %] [% aufs.num_1st %] [% aufs.num_2nd %] min-size=[% aufs.min_size %]<br />
[% END -%]<br />
quick_abort_min 32 KB<br />
quick_abort_max 32 KB<br />
quick_abort_pct 95<br />
store_dir_select_algorithm round-robin<br />
cache_replacement_policy lru<br />
cache_swap_low [% swap_low %]<br />
cache_swap_high [% swap_high %]<br />
#################log#######################################<br />
logformat apache_like %tl %6tr %>a %Ss/%03Hs %<st %rm %ru %Sh/%<A %mt “%{Referer}>h” “%{User-Agent}>h”<br />
access_log [% access_log %] [% logformat %]<br />
cache_log [% cache_log %]<br />
cache_store_log none<br />
logfile_rotate 4<br />
strip_query_terms off<br />
#################configs###################################<br />
[%# 这里就是使用WARPPER特别的一点,必须用content标签标记插入位置 %]<br />
[% content %]<br />
http_reply_access allow all<br />
refresh_pattern -i .tar 180 20% 10080 override-expire ignore-reload reload-into-ims<br />
##########ACL2###################<br />
acl Safe_ports port 80<br />
acl manager proto cache_object<br />
acl ControlCenter src 127.0.0.1<br />
acl PURGE method PURGE<br />
http_access allow Safe_ports<br />
http_access allow PURGE ControlCenter<br />
http_access allow manager ControlCenter<br />
http_access deny PURGE !ControlCenter<br />
http_access deny all<br />
#############snmp############################<br />
acl snmppublic snmp_community cacti_china<br />
snmp_access allow snmppublic ControlCenter<br />
snmp_access deny all<br />
always_direct allow all<code class="highlighter-rouge">
最后域名配置config-china.com.yml如下:
</code>yaml—<br />
#yaml格式,用” “区分层次,用”: “区分hash,用”- “区分array<br />
cache_deny_list: <br />
- “^http://www.china.com/”<br />
- “^http://bbs.china.com/.</em>.html”<br />
custom: china<br />
http_access_list: <br />
#下面两个-,第一个是优先级的数组标示,第二个是同一优先级里多条acl的数组标示<br />
- <br />
- <br />
access: deny<br />
priority: 9<br />
url_regex: “^http://www.china.com/index.html”<br />
- <br />
access: deny<br />
priority: 9<br />
url_regex: “^http://news.china.com/.<em>.htm”<br />
#嗯,上面优先级为9的数组元素里有两个acl,下面优先级为8和7的数组元素里都只有一个acl<br />
- <br />
- <br />
access: allow<br />
priority: 8<br />
url_regex: “^http://.</em>.china.com/.<em>.html”<br />
- <br />
- <br />
access: deny<br />
allow_referer: <br />
- china.com<br />
- cdc.com<br />
deny_info: http://dvs.china.com/do_not_delete.png<br />
priority: 7<br />
url_regex: ‘^http://img.china.com/.</em>.jpg$’<br />
refresh_patterns: <br />
- <br />
max: 1440<br />
min: 180<br />
options: <br />
- ignore-reload<br />
- reload-into-ims<br />
per: 20<br />
url_regex: ‘^http://.*china.com/.+.(jsp|do)’<code class="highlighter-rouge">
另一个配置config-cdcgame.net.yml如下:
</code>yaml<br />
custom: cdcgame<br />
rewrite:<br />
concurrency: 5<br />
program: /usr/local/squid/bin/rewrite.pl<br />
url_regex: ‘^http://www.cdcgame.net/[0-9]+.js\?’<code class="highlighter-rouge">
主要解决的就是acl和http_access的配合问题,最后想是通过优先级数组的方式,同一优先级的acl写完后就先写对应的http_access;这样yml书写起来有些啰嗦,最好还是能有web页面~~
最后运行命令"perl tt4squid.pl config.tt",结果如下:
</code>squid#################ACL1############################<br />
acl all src 0.0.0.0/0.0.0.0<br />
#############################################<br />
http_port 80 accel vhost vport http11 allow-direct<br />
icp_port 0<br />
acl shoutcast rep_header X-HTTP09-First-Line ^ICY.[0-9]<br />
upgrade_http0.9 deny shoutcast<br />
negative_ttl 120 second<br />
refresh_stale_hit 0 minute<br />
vary_ignore_expire on<br />
acl apache rep_header Server ^Apache<br />
broken_vary_encoding allow apache<br />
cache_vary on<br />
cache_mgr admin@test.com<br />
visible_hostname bja-01.test.com<br />
icp_access deny all<br />
cache_effective_user nobody<br />
cache_effective_group nobody<br />
httpd_suppress_version_string on<br />
debug_options ALL,1<br />
#####################################<br />
pipeline_prefetch on<br />
pid_filename /var/run/squid.pid<br />
hierarchy_stoplist aspx cgi \?<br />
######################################<br />
cache_mem 512 MB<br />
maximum_object_size_in_memory 56 KB<br />
maximum_object_size 8 MB<br />
minimum_object_size 0 KB<br />
cache_dir coss /coss 1000000 max-size=8000000 block-size=8000 membufs=512<br />
cache_dir coss /coss2 1000000 max-size=8000000 block-size=8000 membufs=512<br />
cache_dir aufs /aufs 1000000 128 128 min-size=8000000<br />
quick_abort_min 32 KB<br />
quick_abort_max 32 KB<br />
quick_abort_pct 95<br />
store_dir_select_algorithm round-robin<br />
cache_replacement_policy lru<br />
cache_swap_low 70<br />
cache_swap_high 85<br />
#################log#######################################<br />
logformat apache_like %tl %6tr %>a %Ss/%03Hs %<st %rm %ru %Sh/%<A %mt “%{Referer}>h” “%{User-Agent}>h”<br />
access_log /data/proclog/squid/access_log apache_like<br />
cache_log /data/proclog/squid/cache_log<br />
cache_store_log none<br />
logfile_rotate 4<br />
strip_query_terms off<br />
#################configs###################################</p>
<p>####cdcgame<br />
acl cdcgame_url_rewrite url_regex -i ^http://www.cdcgame.net/[0-9]+.js\?<br />
url_rewrite_access deny !cdcgame_url_rewrite<br />
url_rewrite_program /usr/local/squid/bin/rewrite.pl<br />
url_rewrite_concurrency 5</p>
<p>####china<br />
acl no_cache_acl4china url_regex -i ^http://www.china.com/<br />
acl no_cache_acl4china url_regex -i ^http://bbs.china.com/.<em>.html<br />
cache deny no_cache_acl4china<br />
acl acl_china_deny_9 url_regex -i ^http://www.china.com/index.html<br />
acl acl_china_deny_9 url_regex -i ^http://news.china.com/.</em>.htm<br />
http_access deny acl_china_deny_9<br />
acl acl_china_allow_8 url_regex -i ^http://.<em>.china.com/.</em>.html<br />
http_access allow acl_china_allow_8<br />
acl acl_china_deny_7 url_regex -i ^http://img.china.com/.<em>.jpg$<br />
http_access deny acl_china_deny_7<br />
acl not_null_referer referer_regex -i .<br />
acl china_allow_referer referer_regex -i china.com cdc.com<br />
http_access allow acl_china_deny_7 !not_null_referer<br />
http_access deny acl_china_deny_7 china_allow_referer<br />
refresh_pattern -i ^http://.</em>china.com/.+.(jsp|do) 180 20% 1440 ignore-reload reload-into-ims</p>
<p>http_reply_access allow all<br />
…(略)```</p>
blog备份脚本
2011-10-24T00:00:00+08:00
perl
perl
mysql
http://chenlinux.com/2011/10/24/backup-script-4-my-blog
<p>之前总不重视自己的博客,上回一丢才心疼,现在重视起来,决定定期备份sql。写个小脚本如下:<br />
<code class="highlighter-rouge">perl#!/usr/bin/perl
use warnings;
use strict;
use MySQL::Backup;
use Mail::Sender;
open my $tmp_sql, '>', "backup.sql";
my $mb = new MySQL::Backup('dbname', 'localhost', 'dbuser', 'dbpasswd', {'USE_REPLACE' => 1, 'SHOW_TABLE_NAMES' => 1});
print $tmp_sql $mb->create_structure();
print $tmp_sql $mb->data_backup();
close $tmp_sql;
my $sender = new Mail::Sender { smtp => 'smtp.163.com',
from => 'mailuser@163.com',
# debug => 'backup_debug.log',
auth => 'LOGIN',
authid => 'mailuser',
authpwd => 'mailpasswd',
};
$sender->MailFile({ to => 'mailuser@gmail.com',
subject => 'Backup Blog SQL_'.time(),
msg => '3Q',
file => 'backup.sql',});</code><br />
没有直接用mysqldump,而是找了这个MySQL::Backup模块,试着看了导出的sql,和mysqldump的结果是有些不同的。<br />
mysqldump导出的sql一般结构是这样子:<br />
<code class="highlighter-rouge">mysqlDROP TABLE IF EXISTS `tablename`;
CREATE TABLE `tablename`(ID INT NOT NULL ...);
LOCK TABLES `tablename` WARITE;
INSERT INTO `tablename` VALUES(...),(...),(...);
UNLOCK TABLES;</code><br />
而MySQL::Backup导出的sql结构是这样子的:<br />
<code class="highlighter-rouge">mysqlCREATE TABLE `tablename`(ID INT NOT NULL ...);
REPLACE INTO `tablename`(ID,...)VALUES(1,...);
REPLACE INTO `tablename`(ID,...)VALUES(2,...);</code><br />
其实我不太清楚replace比insert好在那,不过pod上的example用了USE_REPLACE=>’1’,就照抄了,如果习惯insert的,在new构建对象时,不用这个param就行了。<br />
另外这个Mail::Sender模块,是在微博上某次评论时,发现很多朋友在用的,我也就放弃一次Net::SMTP_auth,用一次试试,感觉还不错~~</p>
博客恢复了,截图纪念一下
2011-10-20T00:00:00+08:00
http://chenlinux.com/2011/10/20/screenshot-4-cloudhosting-down
<p>这两天折腾博客宕机问题啊~原本对阿里云的看好大幅下降……几次挂掉,甚至电话告知说数据恢复不了请自行重置……崩溃原因居然解释说“因为你没有修改grub再重启”这种解释,拜托,我要重启还不是因为突然ssh报密码错误而你们“重置密码必须重启”?死循环啊!<br />
对了,我人格保证,作为一个用xen当工作环境一年多的运维,绝对没有犯自行升级内核导致重启失败这种错误的可能。阿里云客服能不能不要把一个客户的缘由安到别的客户头上……<br />
不管如何,结局还是好的。截个监控宝的告警图纪念一下:<br />
<img src="/images/uploads/20111020192305.png" alt="" title="aliyunOS" width="856" height="332" class="alignnone size-full wp-image-2647" /></p>
ProBIND体验笔记
2011-10-17T00:00:00+08:00
DNS
php
http://chenlinux.com/2011/10/17/notes-about-probind
<p>闲的无聊,继续研究DNS周边产品,这次盯上了Probind。这是搜索结果中比较常见的bind的web管理工具。其实我是比较希望有个中文的东东方便我偷懒,可惜看sina之前的xbayDNS一直停留在了只支持FreeBSD/MAC的阶段没有更新,汗,不支持linux的项目偶见过的还真不多……<br />
probind最近一次更新也是2003/05/24的事情了。所以也没期待它能多么适应现在的bind体系,不过作为代码看看还是可以的。<br />
创建mysql用户和库,然后把程序解压到webroot目录,然后执行mysql -u named -p named < etc/mktables.sql,这就是install的步骤。但是这个时候访问首页是有一堆报错的,提示你dns服务器的默认配置(外部检测用dns,管理员邮箱等等)没配置。这个需要通过./tools/settings.php去添加——但是首页上没有链接点击,得自己手敲url,汗……<br />
然后,probind更新的时候,估计php还是以version4为主,所以里头用的还是$HTTP_GET_VARS和$HTTP_POST_VARS等全局变量。奇怪的是我把php.ini里的register_global改成On后重启httpd了,页面依然没变,不得已在./inc/lib.inc文件的开头加上了两句<br />
<code class="highlighter-rouge">php$HTTP_GET_VARS = &$_GET;
$HTTP_POST_VARS = &$_POST;</code>才好。<br />
OK,现在正式看到probind的页面了。demo地址:<a href="http://chenlinux.com/probind/">点这里</a><br />
主要就是一个zone的管理,有add、delete、browse三个页面,前两个就是标准的表单,倒是browse里有个test,蛮好玩的,调用bin/testns脚本,使用perl的Net::DNS模块测试zone内的正/反向解析是否正常。<br />
然后就是record的管理,在browse zones里点进zone就可以编辑record了,主要就是主机名、解析ip。<br />
最后是server的管理,这里管理的是真实的DNS的ip和type(master|slave)——probind在易用和性能之间取了一个平衡,他不是像mysqlbind或者mysql-dlz那样直接从db里取数据做响应,而是每次更新(这里区分开了步骤,update的时候只是更新了db,然后再去bulk update的时候才是真正update dns配置)的时候,从数据库生成文件(bin/mkzonefile),再同步到DNS服务器上(sbin/push.local|remote)并执行rndc reconfig命令。</p>
<hr />
<p>每次添加一个server的时候,都会在probind/HOSTS/目录下生成一个同名目录,里面存有从template复制出来的named.tmpl模板,reconfig.sh脚本,rndc.conf和root.hint。<br />
由上可见:<br />
第一,其实probind的管理方式,很像一个简单的集中式配置管理系统;<br />
第二,probind虽然号称支持bind9,但是缺失了现在来看最关键的acl+view体系。不过想到第一点,其实加一个view配置也不是很复杂。大概列一下:<br />
新建views表,包括id、area、iplist和zonefile字段;<br />
修改records表,把关联zones.id的zone改成view关联views.id;<br />
修改inc/lib.inc文件,把add_domain()里的$zonefile命名从$domain.dns改成$domain+$views.id的格式。<br />
修改brzones.php页面,改成先browse views,然后在view里面再选zones;<br />
和template的小小变动。这个可能就多了……</p>
一个ddns的demo
2011-09-30T00:00:00+08:00
perl
http://chenlinux.com/2011/09/30/dynamic-dns-demo
<p>上回分析lbnamed的时候,开玩笑说自己也可以试试在模块基础上加点啥功能。国庆节前最后一天,没啥事情做,就写个小demo续貂。代码如下:<br />
```perl<br />
#!/usr/bin/perl<br />
use warnings;<br />
use strict;<br />
use autodie;<br />
use Sys::Hostname;<br />
use YAML::Syck;<br />
use Net::IP::Match::Regexp qw( create_iprange_regexp match_ip );<br />
use Stanford::DNS;<br />
use Stanford::DNSserver;<br />
$SIG{‘HUP’} = ‘catch_hup’;<br />
my $need_reload;<br />
my $hostmaster = ‘domain.chenlinux.com’;<br />
my $soa = rr_SOA(hostname(), $hostmaster, time(), 3600, 1800, 86400, 0);<br />
my $ns = Stanford::DNSserver->new( listen_on => [ hostname() ],<br />
# debug => 1,<br />
loopfunc => \&conf_reload,<br />
# daemon => ‘no’,<br />
);<br />
my $regexp;<br />
my @domains;<br />
my $arealist;<br />
my $templist;<br />
my $config_path = ‘/data/chenlinux.com/perl/’;<br />
my $ns_domain = ‘test.domain.chenlinux.com’;<br />
$ns->add_dynamic(“$domain” => \&dyn_lb ) foreach my $domain ( @domains );<br />
$ns->add_static( “$ns_domain”, T_SOA, $soa);<br />
$ns->add_static( “$ns_domain”, T_NS, rr_NS($hostmaster));</p>
<p>$ns->answer_queries();</p>
<p>sub catch_hup {<br />
$need_reload = 1;<br />
};</p>
<p>sub conf_reload {<br />
if( $need_reload ) {<br />
load_config();<br />
$need_reload = 0;<br />
};<br />
};</p>
<p>sub load_config {<br />
$regexp = ip2area(“ip.list”);<br />
@domains = grep {s/${config_path}config-(.+?).yml/$1/} glob(“${config_path}*”);<br />
foreach my $domain ( @domains ) {<br />
$arealist->{“$domain”} = LoadFile(“${config_path}config-${domain}.yml”);<br />
@{$templist->{“$domain”}->{“$<em>”}} = @{$arealist->{“$domain”}->{“$</em>”}->{‘per’}} foreach keys %{$arealist->{“$domain”}};<br />
};<br />
};</p>
<p>sub dyn_lb {<br />
my ($domain,$residual,$qtype,$qclass,$dm,$from) = @_;<br />
my $ttl = 3600;<br />
my $ip = area2resolv($domain, $from);<br />
$dm->{‘answer’} .= dns_answer(QPTR, T_A, C_IN, $ttl, rr_A($ip));<br />
$dm->{‘ancount’} += 1;<br />
return 1;<br />
};</p>
<p>sub ip2area {<br />
my $file = shift;<br />
my $area = {};<br />
my $last_area;<br />
open my $fh, ‘<’, $file;<br />
while(<$fh>){<br />
if ( /^acl (\w+)/ ) {<br />
$last_area = $1;<br />
} elsif ( /^((\d{1,3}.?){4});/ ) {<br />
$area->{“$1”} = $last_area;<br />
} else {<br />
next;<br />
};<br />
};<br />
my $regexp = create_iprange_regexp($area);<br />
return $regexp;<br />
};</p>
<p>sub area2resolv {<br />
my $from = shift;<br />
my $area = match_ip( “$from”, $regexp );<br />
my $ip;<br />
my $len = $#{$arealist->{“$domain”}->{“$area”}->{‘per’}};<br />
for ( 0 .. $len ) {<br />
if ( $arealist->{“$domain”}->{“$area”}->{‘per’}->[$<em>] ) {<br />
$ip = $arealist->{“$domain”}->{“$area”}->{‘ip’}->[$</em>];<br />
$arealist->{“$domain”}->{“$area”}->{‘per’}->[$<em>]–;<br />
last;<br />
};<br />
if ( $</em> == $len ) {<br />
@{$arealist->{“$domain”}->{“$area”}->{‘per’}} = @{$templist->{“$domain”}->{“$area”}};<br />
redo;<br />
};<br />
};<br />
return ip_conv(“$ip”);<br />
};</p>
<p>sub ip_conv {<br />
my $ip = shift;<br />
return ($1«24)|($2«16)|($3«8)|$4 if $ip =~ m/(\d+).(\d+).(\d+).(\d+)/;<br />
}<code class="highlighter-rouge">
其中调用的ip.list是bind9用的acl格式,即:
</code>bash<br />
acl cnc_beijing {<br />
202.106.0.0/24;<br />
…<br />
}<br />
…<code class="highlighter-rouge">
这种格式。
调用的config-www.domain.com.yml是YAML格式定义的地区指向ip,即:
</code>yaml<br />
ctc_hebei:<br />
ip:<br />
- 10.168.168.1<br />
- 10.168.169.2<br />
- 10.168.170.3<br />
per:<br />
- 50<br />
- 30<br />
- 20<br />
```<br />
超级简单(其实是我没想到好的weight实现方式)的算法,就是找到这个ctc_hebei的时候,依次序返回ip,同时每返回一次对应的per就减1,减到0就换下一个ip,都0了就复原从头开始。</p>
<p>严重缺失的地方:<del datetime="2011-10-13T05:40:51+00:00">读取不同域名配置;</del>对server的监控;<del datetime="2011-10-13T05:40:51+00:00">对config.yml的reload</del>。</p>
写个同步分发系统(三)
2011-09-29T00:00:00+08:00
dancer
perl
http://chenlinux.com/2011/09/29/dancing-website-for-sync-dist-3
<p>上篇写的页面上,留下一个超链接,查看每条任务的具体情况。现在完成这部分。<br />
首先修改数据库结构,上篇已经建了websync.websync_peer表,现在继续:<br />
```mysql<br />
create table websync_customer (<br />
uid int not null auto_increment primary key,<br />
user varchar(20) not null,<br />
passwd char(32) not null,<br />
custom_info varchar(128),<br />
node varchar(128) not null<br />
) engine=innodb;</p>
<p>create table remote_node (<br />
nid int not null auto_increment primary key,<br />
node_name varchar(16) not null,<br />
node_ip int(16) not null<br />
) engine=innodb;</p>
<p>create table task_msg (<br />
id int not null auto_increment primary key,<br />
task_id int not null,<br />
node_id int not null,<br />
node_md5 char(32) default null,<br />
key (task_id),<br />
key (node_id),<br />
constraint task_f foreign key task_id references websync_peer (id) on delete cascade on update cascade,<br />
constraint node_f foreign key node_id references remote_node (nid) on delete cascade on update cascade,<br />
) engine=innodb;<code class="highlighter-rouge">
主要内容,一是每个用户使用多少节点;二是各节点下载完url后反馈的md5值。
然后新增dancer动作如下:
</code>perlget ‘/checkstatus’ => sub {<br />
my $task_id = params->{‘id’};<br />
my $user = session->{‘login’};<br />
my @status;</p>
<div class="highlighter-rouge"><pre class="highlight"><code>my $task_sth = database->prepare('select md5_hex from websync_customer where id = ?');
$task_sth->execute( $task_id );
my $peer_md5 = $task_sth->fetchrow_hashref->{'md5_hex'};
my $node_sth = database->prepare('select node from websync_customer where user = ?');
$node_sth->execute( $user );
my $nodes = $node_sth->fetchrow_hashref->{'node'};
my $check_sql = 'selct remote_node.node_name node, task_msg.node_md5 md5 from task_msg join remote_node on (task_msg.node_id = remote_node.nid) where task_msg.task_id = ? and task_msg.node_id in ( ? )';
my $check_sth = database->prepare( $check_sql );
$check_sth->execute( $task_id, $nodes );
while ( my $ref = $check_sth->fetchrow_hashref ) {
my $node_name = $ref->{'node'};
my $node_result;
if ( ! defined $peer_md5 ) {
$node_result = 'Peer synchronizing';
} elsif ( ! defined $ref->{'md5'} ) {
$node_result = 'Remote distributing';
} elsif ( $ref->{'md5'} == $peer_md5 ) {
$node_result = 'Distribute Over';
} else {
$node_result = 'Distribute Not Match';
};
push @status, { name => $node_name, result => $node_result, };
};
template 'checkstatus', { 'status' => \@status, }; }; ``` 对应的TT模板如下: ```html<span class="nt"><html></span>
</code></pre>
</div>
<head></head>
<body><table>
<tr><th>TASK</th><td><% task %></td></tr>
<tr><th>NODE</th><th>STATUS</th></tr>
<% FOREACH node IN status %>
<tr><td><% node.name %></td>
<td><% node.result %></td></tr>
<% END %>
</table></body>
<p></html><code class="highlighter-rouge">
然后需要给admin加一个管理页面,勾选恰当的节点分配给客户。动作配置如下:
</code>perlany [‘get’, ‘post’] => ‘/nodeadd’ => sub {<br />
if ( request->method() eq ‘GET’ ) {<br />
my $node_sth = database->prepare(‘select node_name,nid from remote_node order by nid’);<br />
$node_sth->execute();<br />
my @nodes;<br />
while ( my $ref = $node_sth->fetchrow_hashref ){<br />
push @nodes, { name => $ref->{‘node_name’}, id => $ref->{‘nid’}, };<br />
};</p>
<div class="highlighter-rouge"><pre class="highlight"><code> my $user_sth = database->prepare('select user from websync_customer');
$user_sth->execute();
my @users;
while (my $ref = $user_sth->fetchrow_hashref ) {
push @users, $ref->{'user'};
};
template 'nodeadd', { 'users' => \@users,
'nodes' => \@nodes,
};
} else {
my $user = params->{'user'};
my $nodes = params->{'nodes'};
my $add_sth = database->prepare('update remote_node set nodes = ? where user = ?');
$add_sth->execute( $nodes, $user );
}; };``` 对应的TT模板如下: ```html<span class="nt"><html></span>
</code></pre>
</div>
<head>
<script type="text/javascript">
function post_node() {
var nodes = '';
var user = '';
$("form > [type=checkbox]").each(function(){
if($(this)[0].checked) {
nodes += $(this).val()+',';
}
});
$("select > option").each(function(){
if($(this).attr('selected')==true) {
user = $(this).val();
}
});
$.post('/nodeadd?nodes='+nodes+'&user='+user);
}
</head><body>
<form method="post" action="post_node()">
<% FOREACH node IN nodes %>
<input type="checkbox" name="node" value="<% node.id %>" /><% node.name %>
<% END %>
<HR />
<select name="customer">
<% FOREACH user IN users %>
<option value="<% user %>"><% user %></option>
<% END %>
<input type="submit" value="submit">
</form>
</body>
</html>```
从度娘那抄了个jquery例子来用……
</script></head>
写个同步分发系统(二)
2011-09-26T00:00:00+08:00
dancer
perl
http://chenlinux.com/2011/09/26/dancing-website-for-sync-dist-2
<p>接上篇,加上分发过程查看的页面。这应该是一个很典型的翻页处理。<br />
首先创建一个数据库表如下:<br />
<code class="highlighter-rouge">mysqlcreate table websync_peer (
id int not null auto_increment primary key,
begin_time timestamp not null default 0,
end_time timestamp on update current_timestamp,
url varchar(128) not null,
customer varchar(20) not null,
md5_hex char(32) default null
) engine=innodb;</code><br />
然后把之前的peer_query()函数修改如下:<br />
<code class="highlighter-rouge">perlsub peer_query {
my $url = shift;
#这里的database和session都需要其他plugin的配合,见之前博客,不贴重复代码了
my $sth = database->prepare('insert into websync_peer (begin_time, url, customer) value (now(), ?, ?)');
$sth->execute($url, session->{login});
};</code><br />
然后把gearman::client的功能改到mysql的UDFs内完成,做法见<a href="http://www.php-oa.com/2010/09/20/perl-gearman-server-mysql-udfs.html" title="http://www.php-oa.com/2010/09/20/perl-gearman-server-mysql-udfs.html"></a>。<br />
然后写翻页函数了~<br />
<code class="highlighter-rouge">perlget '/check' => sub {
my $from = params->{page} || 1;
my $user = session->{login};
my @urls;
my $count_sql = 'select count(id) count from websync_peer where customer = ?';
my $count_sth = database->prepare($sql);
$count_sth->execute( $user );
my $count = $count_sth->fetchrow_hashref->{count};
my $total_pages = int( $count / 20 + 1 );
return 'No url has been posted to purge.' unless $count;
return 'Selected page number out of range.' if $from > $total_pages;
my $url_sql = 'select id,url,begin_time from websync_peer where customer = ? order by id desc limit ?, 20';
my $url_sth = database->prepare($sql);
$url_sth->execute( $user, ($from - 1) * 20 );
while ( my $ref = $sth->fetchrow_hashref ) {
push @urls, $ref;
};
template 'check', { 'urls' => \@urls,
'prev' => $from > 1 ? $from - 1 : 1,
'next' => $from < $total_pages ? $from + 1 : $total_pages,
'last' => $total_pages,
};
};</code><br />
对应的check.tt如下:<br />
```html<html><head></p>
<style type="text/css">
#main {text-align:center;width:500px;margin-right:auto;margin-left:auto;padding:0px;}
#url {width:776px;border:1px;}
#page_link ul {list-style:none;}
#page_link li {float:left;width:100px;margin-left:3px;line-height:30px;}
</style>
<p></head><body></p>
<div id="main">
<div id="url">
<table>
<tr><th>ID</th><th>URL</th><th>TIME</th><th>MORE</th></tr>
<% FOREACH ref IN urls %>
<tr>
<td><% ref.id %></td>
<td><% ref.begin_time %></td>
<td><% ref.url %></td>
<td><a href="/checkstatus?id=<% ref.id %>">more</a></td>
</tr>
<% END %>
</table>
</div>
<div id="page_link"><ul>
<li><a href="/check?page=1">1</a></li>
<li><a href="/check?page=<% prev %>"><% prev %></a></li>
<li><a href="/check?page=<% next %>"><% next %></a></li>
<li><a href="/check?page=<% last %>"><% last %></a></li>
</ul></div>
</div>
<p></body></html>```<br />
额,傻乎乎的css,好难看,好难写啊……<br />
下一步继续修改mysql表结构,然后完成页面里提供的/checkstatus功能。</p>
linux上获取本机ip的各种perl写法
2011-09-20T00:00:00+08:00
linux
bash
perl
python
ruby
http://chenlinux.com/2011/09/20/get-ip-address-by-perl-on-linux
<p>大家讨论使用 Gearman 做分布式处理时,各机需要注册一个独立的 job 作为信息反馈,但是为了方便,<code class="highlighter-rouge">Gearman::Worker</code> 脚本 <code class="highlighter-rouge">register_function</code> 代码又要通用,于是想到了使用各自的 ip 地址作为 job 命名~</p>
<p>那么怎么在 worker 脚本里获取本机 ip 作为 func 呢?</p>
<ul>
<li>第一种办法,最简单的,调用 shell:</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">$ip</span> <span class="o">=</span> <span class="sb">`ifconfig eth0|grep -oE '([0-9]{1,3}\.?){4}'|head -n 1`</span><span class="p">;</span>
</code></pre>
</div>
<p>注:这里输入是固定的,所以简单的 <code class="highlighter-rouge">[0-9]{1,3}</code> 了,如果是在 web 程序等地方验证 ip,需要更严谨!</p>
<p>或者</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">$ip</span> <span class="o">=</span> <span class="sb">`ifconfig eth0|awk -F: '/inet addr/{split($2,a," ");print a[1];exit}'`</span><span class="p">;</span>
</code></pre>
</div>
<p>好吧,这样显得太不 perl 了,而且频繁的调用外部 shell 不太好</p>
<ul>
<li>第二种:</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">open</span> <span class="nv">FH</span><span class="p">,</span><span class="s">"ifconfig eth0|"</span><span class="p">;</span>
<span class="k">while</span><span class="p">(</span><span class="sr"><FH></span><span class="p">){</span>
<span class="k">last</span> <span class="k">unless</span> <span class="sr">/inet addr:((\d{1,3}\.?){4})/</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">$1</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
<p>看起来稍微 perl 了一些,虽然实质跟上面的调用 shell 和 grep 法是一样的。</p>
<ul>
<li>第三种,更 perl 一点,纯粹读文件:</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nb">open</span> <span class="nv">FH</span><span class="p">,</span><span class="s">'<'</span><span class="p">,</span><span class="s">'/etc/sysconfig/network-scripts/ifcfg-eth0'</span><span class="p">;</span>
<span class="k">while</span><span class="p">(</span><span class="sr"><FH></span><span class="p">){</span>
<span class="k">next</span> <span class="k">unless</span> <span class="sr">/IPADDR\s*=\s*(\S+)/</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">$1</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
<p>进一步的,如果不一定 rh 系,还要去读 <code class="highlighter-rouge">/etc/issue</code> ,确定网络配置文件到底是 <code class="highlighter-rouge">/etc/sysconfig/network-script/ifcfg-eth0</code> 还是 <code class="highlighter-rouge">/etc/network/interfaces</code> 还是其他,然后根据不同发行版写不同的处理方法……额,这是打算自己写模块么?</p>
<p>好吧,大家来充分体会 <code class="highlighter-rouge">CPAN</code> 的魅力,去 search 一下,找到一把 <code class="highlighter-rouge">Sys::HostIP</code>、<code class="highlighter-rouge">Sys::HostAddr</code>、<code class="highlighter-rouge">Net::Inetface</code> 等模块。</p>
<ul>
<li>第四种:</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="k">use</span> <span class="nn">Sys::</span><span class="nv">HostAddr</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$interface</span> <span class="o">=</span> <span class="nn">Sys::</span><span class="nv">HostAddr</span><span class="o">-></span><span class="k">new</span><span class="p">(</span><span class="nv">ipv</span> <span class="o">=></span> <span class="s">'4'</span><span class="p">,</span> <span class="nv">interface</span> <span class="o">=></span> <span class="s">'eth0'</span><span class="p">);</span>
<span class="k">print</span> <span class="nv">$interface</span><span class="o">-></span><span class="nv">main_ip</span><span class="p">;</span>
</code></pre>
</div>
<p>不过进去看看pm文件,汗,这几个模块都是调用ifconfig命令,不过是根据发行版的不同进行封装而已。</p>
<p>还有办法么?还有,看</p>
<ul>
<li>第五种:</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nv">perl</span> <span class="o">-</span><span class="nv">MPOSIX</span> <span class="o">-</span><span class="nv">MSocket</span> <span class="o">-</span><span class="nv">e</span> <span class="s">'my $host = (uname)[1];print inet_ntoa(scalar gethostbyname($host))'</span><span class="p">;</span>
</code></pre>
</div>
<p>不过有童鞋说了,这个可能因为hostname的原因,导致获取的都是127.0.0.1……</p>
<p>那么最后还有一招。通过 <code class="highlighter-rouge">strace ifconfig</code> 命令可以看到,linux 实质是通过 ioctl 命令完成的网络接口 ip 获取。那么,我们也用 <code class="highlighter-rouge">ioctl</code> 就是了!</p>
<ul>
<li>第六种如下:</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="c1">#!/usr/bin/perl</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Socket</span><span class="p">;</span>
<span class="nb">require</span> <span class="s">'sys/ioctl.ph'</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">get_ip_address</span><span class="p">($)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$pack</span> <span class="o">=</span> <span class="nb">pack</span><span class="p">(</span><span class="s">"a*"</span><span class="p">,</span> <span class="nb">shift</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$socket</span><span class="p">;</span>
<span class="nb">socket</span><span class="p">(</span><span class="nv">$socket</span><span class="p">,</span> <span class="nv">AF_INET</span><span class="p">,</span> <span class="nv">SOCK_DGRAM</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
<span class="nb">ioctl</span><span class="p">(</span><span class="nv">$socket</span><span class="p">,</span> <span class="nv">SIOCGIFADDR</span><span class="p">(),</span> <span class="nv">$pack</span><span class="p">);</span>
<span class="k">return</span> <span class="nv">inet_ntoa</span><span class="p">(</span><span class="nb">substr</span><span class="p">(</span><span class="nv">$pack</span><span class="p">,</span><span class="mi">20</span><span class="p">,</span><span class="mi">4</span><span class="p">));</span>
<span class="p">};</span>
<span class="k">print</span> <span class="nv">get_ip_address</span><span class="p">(</span><span class="s">"eth0"</span><span class="p">);</span>
</code></pre>
</div>
<p>这样的好处,就是只调用了核心模块,在分发脚本时,不用连带安装其他模块。</p>
<p><em>注</em>:这个其实是根据网上有的一个 py 的脚本修改的</p>
<ul>
<li>py版如下:</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="c">#!/usr/bin/python</span>
<span class="kn">import</span> <span class="nn">socket</span>
<span class="kn">import</span> <span class="nn">fcntl</span>
<span class="kn">import</span> <span class="nn">struct</span>
<span class="k">def</span> <span class="nf">get_ip_address</span><span class="p">(</span><span class="n">ifname</span><span class="p">):</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">socket</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">socket</span><span class="o">.</span><span class="n">AF_INET</span><span class="p">,</span> <span class="n">socket</span><span class="o">.</span><span class="n">SOCK_DGRAM</span><span class="p">)</span>
<span class="k">return</span> <span class="n">socket</span><span class="o">.</span><span class="n">inet_ntoa</span><span class="p">(</span><span class="n">fcntl</span><span class="o">.</span><span class="n">ioctl</span><span class="p">(</span>
<span class="n">s</span><span class="o">.</span><span class="n">fileno</span><span class="p">(),</span>
<span class="mh">0x8915</span><span class="p">,</span> <span class="c"># SIOCGIFADDR</span>
<span class="n">struct</span><span class="o">.</span><span class="n">pack</span><span class="p">(</span><span class="s">'256s'</span><span class="p">,</span> <span class="n">ifname</span><span class="p">[:</span><span class="mi">15</span><span class="p">])</span>
<span class="p">)[</span><span class="mi">20</span><span class="p">:</span><span class="mi">24</span><span class="p">])</span>
<span class="k">print</span> <span class="n">get_ip_address</span><span class="p">(</span><span class="s">'eth0'</span><span class="p">)</span>
</code></pre>
</div>
<p><em>2012年12月19日增</em>:</p>
<p>为logstash的input/file.rb找到</p>
<ul>
<li>ruby版本的:</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="c1">#!/usr/bin/ruby</span>
<span class="nb">require</span> <span class="s1">'socket'</span>
<span class="no">SIOCGIFADDR</span> <span class="o">=</span> <span class="mh">0x8915</span> <span class="c1"># get PA address </span>
<span class="k">def</span> <span class="nf">get_ip_address</span><span class="p">(</span><span class="n">iface</span><span class="p">)</span>
<span class="k">begin</span>
<span class="n">sock</span> <span class="o">=</span> <span class="no">UDPSocket</span><span class="p">.</span><span class="nf">new</span>
<span class="n">buf</span> <span class="o">=</span> <span class="p">[</span><span class="n">iface</span><span class="p">,</span><span class="s2">""</span><span class="p">].</span><span class="nf">pack</span><span class="p">(</span><span class="s1">'a16h16'</span><span class="p">)</span>
<span class="n">sock</span><span class="p">.</span><span class="nf">ioctl</span><span class="p">(</span><span class="no">SIOCGIFADDR</span><span class="p">,</span> <span class="n">buf</span><span class="p">);</span>
<span class="n">sock</span><span class="p">.</span><span class="nf">close</span>
<span class="n">buf</span><span class="p">[</span><span class="mi">20</span><span class="p">.</span><span class="nf">.</span><span class="mi">24</span><span class="p">].</span><span class="nf">unpack</span><span class="p">(</span><span class="s2">"CCCC"</span><span class="p">).</span><span class="nf">join</span><span class="p">(</span><span class="s2">"."</span><span class="p">)</span>
<span class="k">rescue</span>
<span class="kp">nil</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">if</span> <span class="vg">$0</span> <span class="o">==</span> <span class="kp">__FILE__</span>
<span class="nb">puts</span> <span class="n">get_ip_address</span><span class="p">(</span><span class="s1">'eth0'</span><span class="p">)</span>
<span class="k">end</span>
</code></pre>
</div>
<p>不过看puppet里还是用ifconfig的方法。</p>
写个同步分发系统(一)
2011-09-16T00:00:00+08:00
dancer
perl
http://chenlinux.com/2011/09/16/dancing-website-for-sync-dist-1
<p>写程序这个事情,其实规划最麻烦。比方我其实并没留意过一个完整的同步分发系统都有哪些功能。权做练手,想到哪些写哪些好了。<br />
最基础的部分:一个提交文件列表的页面,自动分析文件列表然后从源下载文件,中心下载完成后通知边缘节点开始;<br />
其次:文件完整性的校验,同步和分发的进度报表页面,失败报表的重试选择和报警;<br />
再次:系统分用户,不同用户可选节点指定分发,只查看当前用户的报表。<br />
以上。<br />
今天先完成最基础的部分。还是用dancer -a websync创建应用。然后创建views/websync.tt如下:<br />
```html<br />
</head><body></p>
<form name="urllist" action="/websync" method="post">
<table border="1" align="center">
<tr><td><% message %><% FOREACH url IN errurls %><% url %><br /><% END %></td></tr>
<tr><td><textarea name="urllist" cols="64" rows="10"></textarea></td></tr>
<tr><td align="center"><input type="submit" name="submit" value="submit" />
<input type="reset" name="cancel" value="cancel" />
</td></tr>
</table>
</form>
<p></body></html><code class="highlighter-rouge">
一如既往的难看……考虑是不是学下用dreamweaver画个稍微好看点的页面出来做layout啊……
然后是lib/websync.pm,如下:
</code>perlpackage websync;<br />
use Dancer ‘:syntax’;<br />
use Gearman::Client;</p>
<p>our $VERSION = ‘0.1’;<br />
#Don’t change index because there are so many otherthings to do! <br />
get ‘/’ => sub {<br />
template ‘index’;<br />
};</p>
<p>any [‘get’, ‘post’] => ‘/websync’ => sub {<br />
my @errurls;<br />
my $message = ‘Write urllist under here.<br />Attention: The url format must like “http://img.domain.com/path/to/example.flv”’;<br />
if ( request->method() eq ‘POST’ ) {<br />
my $url_pattern = qr(^http://[^/]+?.\w+/);<br />
my @urllists = split ‘ ‘, params->{urllist};<br />
foreach ( @urllists ) {<br />
push @errurls, $_ and next unless m/$url_pattern/;<br />
peer_query($_);<br />
};<br />
$message = ‘Sync begin, waiting please.<br />And there are some error urls. Please check them:<br />’;<br />
};<br />
template ‘websync’, { ‘message’ => $message, <br />
‘errurls’ => \@errurls, <br />
};<br />
};</p>
<p>sub peer_query {<br />
my $url = shift;<br />
my @job_servers = qw(127.0.0.1:7003 192.168.0.2:7004);<br />
my $client = Gearman::Client->new;<br />
$client->job_servers(@job_servers);<br />
$client->dispatch_background(‘websync’, $url);<br />
};</p>
<p>true;<code class="highlighter-rouge">
嗯,这里试着用了gearman而不是fork,一个是考虑到可能web系统跟中心存储不在一起;另一个是考虑之后需要用mysql存储分发状态,可以把gearman::client改成mysql的trigger形式。
然后是worker.pl,运行在中心存储上,接受job,完成下载,然后通知其他节点继续:
</code>perl#!/usr/bin/perl -w<br />
use Gearman::Worker;<br />
use LWP::Simple;<br />
use Net::SSH::Perl;<br />
use POSIX ‘:WNOHANG’;<br />
$SIG{CHLD} = sub {waitpid(-1,WNOHANG)};</p>
<p>my @job_servers = qw(127.0.0.1:7003 192.168.0.2);<br />
my $worker = Gearman::Worker->new;<br />
$worker->job_servers(@job_servers);<br />
$worker->register_function( websync => \&websync );<br />
$worker->work while 1;</p>
<p>sub websync {<br />
my $job = shift;<br />
my @path = split(‘/’, $job->arg);<br />
my $filepath = ‘/var/www/’;<br />
foreach ( 2 .. $#path - 1 ) {<br />
$filepath .= $path[$_].’/’;<br />
mkdir $filepath unless -d $filepath; <br />
};<br />
sync_get($job->arg, $filepath . $path[-1]);<br />
};</p>
<p>sub sync_get {<br />
my ( $url, $file ) = @<em>;<br />
my $http_code = getstore($url, $file);<br />
dist($file) if $http_code =~ m/^2/;<br />
};<br />
#I will rewrite this function to use gearman too~<br />
sub dist {<br />
my $file = shift;<br />
my @remote = qw(1.1.1.1 2.2.2.2);<br />
foreach(@remote){<br />
unless(fork){<br />
my $ssh = Net::SSH::Perl->new($</em>);<br />
$ssh->login(root, passwd);<br />
$ssh->cmd(“rsync 192.168.0.2:$file $file”);<br />
};<br />
};<br />
};```<br />
不过想到,其实可以在remote上设定每15分钟一次rsync。这样节省掉中心的dist功能,改成remote上的rsync后,主动通过mysql汇报更新的list和md5。<br />
明天开始改这种方式。</p>
<hr />
<p>晚饭回来,增加dancer在nginx上的部署方式。之前写过apache上用mod_perl的方式,这回因为正好电脑上有nginx,就改用nginx反代了:<br />
首先安装一个perl的server,命令如下:<br />
<code class="highlighter-rouge">bash# cpanm Plack Starman</code><br />
Starman是一个提供prefork方式运行的HTTP服务器。另外还有基于AnyEvent的Twiggy和基于Coro的Corona,不够因为我是在本机的colinux上做实验,装的是UBUNTU9.04系统,已经没有apt源装openssl了,所以Net::SSLeay模块无法安装,AnyEvent类型的也就不能用了。<br />
启动命令如下:<br />
<code class="highlighter-rouge">bashsudo -u www-data plackup -E production -s Starman --workers=2 -l /tmp/plack.sock -a /var/www/websync/bin/app.pl &</code><br />
该命令指定了运行用户,运行server核心,读取的配置文件,启动的worker进程,提供的socket接口。<br />
然后就可以利用nginx的upstream功能,pass到这个socket接口上了。nginx.conf相关部分如下:<br />
```nginx upstream backendurl {<br />
server unix:/tmp/plack.sock;<br />
}</p>
<div class="highlighter-rouge"><pre class="highlight"><code>server {
listen 80;
server_name dancer.test.com;
access_log logs/dancer_access.log;
error_log logs/dancer_err.log info;
root /var/www/websync/public;
location / {
try_files $uri @proxy;
access_log off;
expires max;
}
location @proxy {
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://backendurl;
}
}```
</code></pre>
</div>
lbnamed代码浅读
2011-09-08T00:00:00+08:00
perl
http://chenlinux.com/2011/09/08/lbnamed-code-reading
<p>群里听说lbnamed这么个东东,用perl实现的动态DNS服务器。程序包括三个perl模块:Stanford::DNSserver模块、Stanford::DNS模块、LBCD模块;三个perl程序:lbnamed主程序、poller探测程序、slbcd监控程序(这个有C语言的版本)。</p>
<ul>
<li>先看主程序lbnamed。</li>
</ul>
<p>排除掉sig啊,help啊,pid啊,log啊之类的以后,剩下的主要内容包括:</p>
<ol>
<li>一个Stanford::DNSserver->new()对象,分别用add_static和add_dynamic方法加入的DNS记录,然后用answer_queries方法启动运行。</li>
<li>一个handle_lb_request()函数,用在add_dynamic中完成动态DNS解析响应。</li>
<li>一个load_config()函数,用于从文件中读取domain/ip/weight的值并存入hash。</li>
<li>一个by_weight()函数,用于排序。</li>
</ol>
<ul>
<li>接下来主要是去看Stanford::DNSserver里的add_*和answer_queries函数。</li>
</ul>
<p>这个模块里,$self除了new()传递的参数以外,还有几个很重要的空间,分别是$self->{select}、$self->{static}->{$domain}->{$type}->{answer}、$self->{static}->{$domain}->{$type}->{ancount}和$self->{dynamic}->{$domain}。<br />
1. add_static()很简单,把传入的配置按照格式组装给Stanford::DNS::dns_answer(),返回值存入$self->{static}->{$domain}->{$type}->{answer},同时$self->{static}->{$domain}->{$type}->{ancount}加1。</p>
<pre><code>
*** 2011-10-09注: 代码是.=,即返回值是接入,static方法可以返回多条记录,而dynamic方法只会返回最后加载的一条。
</code></pre>
<ol>
<li>add_dynamic()更简单,直接把lbnamed里的handle_lb_request()的引用存入$self->{dynamic}->{$domain}就完了。</li>
<li>answer_queries()方法启动服务器,步骤如下:</li>
</ol>
<p>$self->daemon(),其实就是改变STD后的fork。<br /><br />
$self->init(),首先用IO::Socket::INET模块,new出两个socket,分别监听在TCP/UDP的DNS端口上;然后调用$self->{select},这在new的时候已经创建了一个IO::Select对象。把之前的两个socket加入到select队列中。<br /><br />
在一个while(1){}死循环中,调用select的can_read()方法,具体是$self->{select}->can_read(600),根据IO::Select的说明,这个600表示如果一直没有可以READ的SOCKET,等待600秒后返回一个空队列,也就是说轮训的时候,每个socket会阻塞600秒~<br /><br />
如果can_read成立,根据其协议是UDP还是TCP,调用$self->handle_udp_req或者$self->handle_tcp_req处理这个socket。</p>
<ul>
<li>然后看$self->handle_udp_req()方法。</li>
</ul>
<ol>
<li>用perl内嵌的recv方法,从socket中读取8192字节到$buff,至于这里为什么是8192字节,没有找到比较合理的解释。因为DNS协议的UDP传输一般只有512字节,而UDP协议的不可靠性,也很难保证8192字节的数据完整。唯一在一篇博客上看到说因为NFS的读写数据大小是这个。反正不管怎么样,8192肯定是足够的;</li>
<li>把$buff传递给$self->do_dns_request()函数,返回值赋为$reply。</li>
<li>用send方法,把$reply发回给socket。</li>
</ol>
<ul>
<li>再来看$self->handle_tcp_req()方法。</li>
</ul>
<ol>
<li>用IO::Socket::INET::accept()方法创建一个返回到对端的socket对象;</li>
<li>关闭原来的socket,释放IO::Select;</li>
<li>使用perl内嵌的sysread方法,从socket中读入2个字节到$buff,然后用unpack(‘n’,$buff)的方式解压得到数据长度;</li>
<li>根据计算的长度,继续读入相应长度的数据,并传递给$self->do_dns_request()函数,返回值赋为$reply;</li>
<li>用pack(‘n’,length $reply)打包数据长度,接在数据报文的头部,用send方法发送出去;</li>
<li>依上面三步的操作,循环进行,直到socket内数据全部读取完成。</li>
</ol>
<ul>
<li>然后看$self->do_dns_request()方法。</li>
</ul>
<ol>
<li>先把buff里的前12字节取出来,然后用unpack(‘n6C*’,$buff)解压成RFC1035标准里的包头信息,包括:</li>
</ol>
<pre><code>
$id——用来验证请求和响应匹配的随机数,
$flags——包括$opcode查询类型(正向/反向)、$qr(请求/响应)、$tc(截断?)、$rd(是否递归),
$qdcount——查询的问题个数,
$ancount——响应的结果个数,
$aucount——额,这个在RFC1035里都没发现,
$adcount——附加区域内的响应结果个数。从程序来看,aucount和adcount一直都是0。
</code></pre>
<ol>
<li>然后用Stanford::DNS::dn_expand()解析$buff出$qname,用unpack(‘nn’, substr($buff, $ptr, 4))解析出$qtype和$qclass,这三个变量是RFC1035.4.1.3中定义的请求区内容。</li>
<li>最后,使用$self->check_static($qname,$qtype,$qclass,\%dnsmsg)或$self->check_dynamic($qname,$qtype,$qclass,\%dnsmsg,$from)方法,得到响应内容,然后pack(‘n6’, $id, $flags, $qdcount, $dnsmsg{ancount}, $dnsmsg{aucount}, $dnsmsg{adcount}) . $reply组装完成返回。</li>
</ol>
<pre><code>
*** 2011-10-09注: 代码是if( * or * ),即是先检查check_static结果并返回,只有不存在static的情况下,才会check_dynamic去!
</code></pre>
<p>上面关于pack/unpack语焉不详,实在是自己也不懂……汗!</p>
<ul>
<li>然后看check_static()函数</li>
</ul>
<p>很简单,从前面add_static存入的%{$self->{static}}里取出对应qname的value即可,同时计算一下ancount。</p>
<ul>
<li>接着是check_dynamic()函数</li>
</ul>
<p>跟static不同的是,这里对域名进行了分割然后重拼装,在获取到最先匹配的handler后跳出循环,调用handler处理。举例说:team.www.domain.com这个域名,它先检查是不是存在$self->{dynamic}->{www.domain.com},有就传递(‘www.domain.com’,’team’,…);否则下一步检查$self->{dynamic}->{domain.com},有就传递(‘domain.com’,’team.www’,…);依此类推……<br />
<br />至于Stanford::DNS模块,主要就是拼装各种数据成DNS协议规范的数据格式,在没看懂RFC1035和pack/unpack的用法前,就不写了。</p>
<ul>
<li>回头继续看那个handler函数</li>
</ul>
<ol>
<li>在new的时候,调用了&do_reload()做loopfunc;</li>
<li>do_reload()中调用了load_config(“${poller_results}lb”)读取poller的结果文本;</li>
<li>load_config()中将groups数组作为key,存入了各项数值,weight、ttl、rnd等;</li>
<li>unless ($group = $lb_groups{$qname})这里就是用到了上一步的key,确认域名存在;</li>
<li>by_weight()中,用$weight{}对$host排序;</li>
<li>选出最终答案,$rnd{$qname} ? @$group[int(rand(min($rnd{$qname},$#$group)))] : @$group[0];不过在load_config中,$rnd{$group}=0(另外一个赋值的地方是$rnd{$weight}=$host,很没道理的地方,我都怀疑是不是写反了?),所以肯定是从数组拿第一个,也就是最大的一个了。</li>
</ol>
<ul>
<li>server部分完毕,然后看poller程序:</li>
</ul>
<ol>
<li>大同小异的也是在while(1){…;sleep(120)}中用IO::Select和IO::Socket::INET完成对host的探测;</li>
<li>然后dump_lb()函数里取出各host的探测结果,计算weight,写入文件。</li>
</ol>
<p>这个计算方法主要包括了服务器的登录user数量和loadavg的大小。说实话我不清楚为啥要计算user数量……具体的数据格式,可以看LBCD.pm里注释的C语言typeof struct定义,也可以看slbcd脚本里的pack。如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nv">$reply</span> <span class="o">=</span> <span class="nb">pack</span><span class="p">(</span><span class="s">"nnnnNNNnnnnnCC"</span><span class="p">,</span> <span class="c1"># build the reply</span>
<span class="nv">$version</span><span class="p">,</span> <span class="nv">$id</span><span class="p">,</span> <span class="nv">$op</span><span class="p">,</span> <span class="nv">$status</span><span class="p">,</span>
<span class="nv">$btime</span><span class="p">,</span> <span class="nb">time</span><span class="p">(),</span> <span class="nv">$utime</span><span class="p">,</span>
<span class="nv">$l1</span><span class="p">,</span> <span class="nv">$l5</span><span class="p">,</span> <span class="nv">$l15</span><span class="p">,</span> <span class="nv">$tot</span><span class="p">,</span> <span class="nv">$uniq</span><span class="p">,</span>
<span class="nv">$console</span><span class="p">,</span> <span class="nv">$reserved</span><span class="p">);</span>
</code></pre>
</div>
<ul>
<li>最后总结</li>
</ul>
<p>这个server利用poller调整解析的weight,但并不是我想象中的百分比解析,而是针对每次解析请求时后端服务器的准实时(120秒)负载情况给出最适合的一个。对于全网GSLB,感觉不太适用;对于内部SLB,又感觉不如LVS好用。<br /><br />
不过作为一个DNS框架,如果抛开poller,倒也可以自己再设想一个动态dns来:</p>
<ol>
<li>用Net::IP::Match::Regexp匹配ip列表到area区域——可以注意到,lbnamed中写的handler中没有用上传递进去的$from;</li>
<li>然后$file是area区域到host/weight(100%)的对应,handler中取rand(100)随机数,跟设定的weight比大小,确定具体用哪个ip;</li>
<li>最后改造poller,根据流量,响应时间等,确定一个weight阈值,超标的话就修改$file中得weight大小。</li>
</ol>
用nginx区分文件大小做出不同响应
2011-08-25T00:00:00+08:00
nginx
perl
http://chenlinux.com/2011/08/25/nginx-diff-response-via-filesize
<p>昨晚和前21v的同事聊天,说到我离职后一些技术上的更新。其中有个给某大客户(游戏下载类)的特殊需求设计,因为文件大小差距很大——估计是大版本和补丁的区别——又走的是同一个域名,而squid在响应比较大的文件时,尤其是初次下载的时候,性能比较差,所以拆成两组服务器,squid服务于较小的文件,通过pull方式从peer层获取,nginx服务于较大的文件,通过push方式由peer层分发同步。外部发布域名一律解析到squid服务器组上,请求透传到peer层的nginx,nginx分析这个url的content-length,如果大于阈值,则不返回文件,而是302到nginx服务器组的独立域名下的相应url去。</p>
<p>这里要注意的是,nginx的内部变量里有一个$content-length,是不能用在这里的,官方wiki是这么解释这个变量的:”This variable is equal to line Content-Length in the header of request”。可见,这个变量是请求头的内容,一般见于POST请求用来限定POST信息的长度;而不是我们需要的响应头的内容。</p>
<p>老东家最后是修改了nginx的src完成的功能。不过我想,这里其实可以使用http_perl_module完成的。而且还可以扩展302跳转的功能,把独立域名改成直接通过remote_addr定向到最近IP上。</p>
<p>因为手头没有服务器,以下内容都是凭空想象,看官们注意……<br />
首先是nginx.conf里的配置:<br />
<code class="highlighter-rouge">nginxhttp {
perl_modules perl;
perl_require SizeDiff.pm;
server {
listen 80;
server_name dl.gamedomain.com;
location / {
perl SizeDiff::handler;
}
}
}</code><br />
然后是perl/SizeDiff.pm,如下:<br />
<code class="highlighter-rouge">perlpackage SizeDiff;
use Nginx::Simple;
sub main {
my $self = shift;
my $webroot = '/www/dl.gamedomain.com/'
return HTTP_NOT_ALLOWED unless $self->uri =~ m!^(/.+/)[^/]+$!;
my $file = $webroot . $1 . $self->filename;
my @filestat = stat($file) or return HTTP_NOT_FOUND;
my $filesize = $filestat[7];
if ( $filesize < 8 * 1024 * 1024 ) {
return OK;
} else {
$self->location('http://bigfile.cdndomain.com'.$self->uri);
}
};
1
</code><br />
大体应该就是上面这样。<br />
之前还考虑过如果不是push方式,可以在perl里考虑使用LWP获取header,不过仔细想想:第一,万一源站开启了chunked获取不到content-length呢?第二,就算可以,如果一个文件是1个G,那再去下载这1个G的文件下来,这个perl进程肯定挂了——官方wiki里可是连DNS解析时间都认为太长……也就是说,这个设想不适合在peer层,而是在loadbalance的角色,通过lwp的header结果,小文件upstream到后端的squid,大文件location到另外的nginx。<br />
另一个可改进的地方,就是self->location前面,可以结合Net::IP::Match::Regexp模块或者自己完成的类似功能,来针对self->remote_addr选择最近的服务器组IP,最后返回location(“http://$ip$uri”)这样。</p>
CloudForecast学习笔记(三)
2011-08-18T00:00:00+08:00
perl
http://chenlinux.com/2011/08/18/learning-cloudfarecast-3
<p>第三篇,看web部分。主程序cloudforecast_web里主要就是调用CloudForecast::Web的run()函数,接下来去看CloudForecast::Web。<br />
照例还是加载配置,然后这里主要多了两个accessor:allowfrom和front_proxy。用来定制acl和代理的。从下文可以看到,分别是采用了Plack::Middleware::Access和Plack::Middleware::ReverseProxy两个模块进行控制。<br />
然后是主要部分,通过Plack::Builder建立$app:<br />
1、初始化:”my $app = $self->psgi;”,这里是调用父层”use Shirahata -base;”的psgi()函数完成的。稍后再看这个。<br />
2、包装:取出前面说的allowfrom和front_proxy部分后,代码如下:<br />
<code class="highlighter-rouge">perl
$app = builder {
enable 'Plack::Middleware::Lint';
enable 'Plack::Middleware::StackTrace';
enable 'Plack::Middleware::Static',
path => qr{^/(favicon\.ico$|static/)},
root =>Path::Class::dir($self->root_dir, 'htdocs')->stringify;
$app;
};</code><br />
真实加载顺序是倒序的,先加载Static.pm,用来服务静态文件,意即url路径为^/static/.*的,实际documentroot为./htdocs/;然后加载StackTrace.pm,用于开发调试的时候,向标准输出输出错误跟踪信息;最后是Lint.pm,用于检查请求/响应的格式是否正确。<br />
然后是加载运行,使用Plack::Loader运行上面build出来的$app。方法如下:<br />
<code class="highlighter-rouge">perl
my $loader = Plack::Loader->load(
'Starlet',
port => $self->port || 5000,
host => $self->host || 0,
max_workers => 2,
);
$loader->run($app);</code><br />
主要是两个参数,第一个是用来运行plack的服务器模块名称,常见的有starman/twiggy/corona/perlbal等等,这里写的这个Starlet,是基于HTTP::Server::PSGI模块添加预派生(prefork)/热部署(Server::Starter)/优雅重启等功能的一个服务器模块,原来叫的名字是”Plack::Server::Standalone::Prefork::Server::Starter”(简称PSSPSS)……</p>
<p>然后去看前面说到的Shirahata.pm里的psgi()函数。<br />
这个Shirahata似乎是作者自己完成的一个框架?反正我在cpan上没看到。psgi()里调用build_app()完成主要功能,其中使用了Router::Simple完成route功能,Data::Section::Simple(提取文件中_DATA_下的内容)和HTML::FillInForm::Lite()、Text::Xslate完成template功能,Plack::Request和Plack::Response完成请求响应功能,最终返回一个”$psgi_res;”。<br />
一堆模块没一个看过的……不细究了……</p>
CloudForecast学习笔记(二)
2011-08-18T00:00:00+08:00
perl
http://chenlinux.com/2011/08/18/learning-cloudfarecast-2
<p>接下来看radar部分,也就是探测主程序cloudforecast_radar,其中主要就是调用CloudForecast::Radar中的run()函数。<br />
首先还是惯例,调用ConfigLoader模块加载配置文件;<br />
然后是%SIG信号的定义,用来之后自动重运行的;<br />
然后一个while true死循环:<br />
1、循环里用select(undef,undef,undef,0.5)实现一个0.5秒的sleep;<br />
2、第一个if语句,用来判断是否是父进程,并使用无阻塞的waitpid($pid,WNOHANG)等待子进程完成——即$kid==-1;<br />
3、第二个if语句,用来强制退出死循环,条件是收到%SIG信号;<br />
4、第三个if语句,确认当前时间超过计划中的执行时间(即离上次执行时间最近的整5分钟点),开始执行探测——采用fork()派生子进程。<br />
5、子进程内容是从之前获取的配置文件内,轮询每一台设备,最终调用run_host()函数执行。<br />
然后又是两个if语句,接在while里的last之后,等待子进程全部完成的。</p>
<p>接下来看run_host()函数,其实就是new了一个CloudForecast::Host对象,并调用其run()函数。<br />
这个run()函数,就是根据config里的resource调用相应的CloudForecast::Data::*,最后到CloudForecast::Data里的call_fetch()函数。ok,这个函数上一篇已经看过了。</p>
CloudForecast学习笔记(一)
2011-08-17T00:00:00+08:00
perl
http://chenlinux.com/2011/08/17/learning-cloudfarecast-1
<p>近三天学习cloudforecast,这是一个日本SA写的分布式监控的perl项目。日本的运维水平和perl水平,都让人羡慕啊……<br />
项目介绍:<br />
http://blog.riywo.com/2011/02/27/043646<br />
demo网址:<br />
http://editnuki.info:5000/<br />
下载地址:<br />
https://github.com/kazeburo/cloudforecast<br />
粗略的看了主要文件,主要是用Class::<em>完成的OO,Plack::MiddleWare::</em>完成的web,Gearman::Worker调用Data::*里的具体模块完成对服务器的监控抓取,然后调用RRDs完成监控数据图像的更新,在启动分布式的情况下,则用Gearman::Client传输监控数据给调用RRDs的worker。</p>
<p>第一篇主要记录一下监控数据在服务器上流程,cloudforecast是怎么去抓取数据,怎么传递给rrd的。<br />
不过,先看看Class::*的用法:<br />
在CloudForecast::Data中,有如下一段代码:<br />
<code class="highlighter-rouge">perl
use base qw/Class::Data::Inheritable Class::Accessor::Fast/;
__PACKAGE__->mk_accessors(qw/hostname address details args
component_config _component_instance
global_config/);
__PACKAGE__->mk_classdata('rrd_schema');
__PACKAGE__->mk_classdata('fetcher_func');
__PACKAGE__->mk_classdata('graph_key_list');
__PACKAGE__->mk_classdata('graph_defs');
__PACKAGE__->mk_classdata('title_func');
__PACKAGE__->mk_classdata('sysinfo_func');</code><br />
这里,先用use base()加载两个父类继承关系。然后用Class::Accessor::Fast的mk_accessors方法创建了一堆可读写的变量,这里有另一种写法,看起来更舒服一些:<br />
<code class="highlighter-rouge">perluse Class::Accessor "moose-like";
has hostname => ( is => 'rw', isa => 'Str' );</code><br />
然后是Class::Data::Inheritable的mk_classdata方法创建了一堆可继承的方法。<br />
在CPAN上看到另外有一个模块叫Class::Data::Accessor的,是上面这两个模块的合集,不过作者声明说已经废弃,推荐大家使用Moose了……</p>
<p>现在来跟踪一下fetch_worker的流程:<br />
先看cf_fetcher_worker脚本里new一个worker出来后,执行的是fetcher_worker();</p>
<p>然后看lib/CloudForecast/Gearman/Worker.pm里的fetcher_worker(),在连接上gearmand上的fetcher任务后,执行的是$self->load_resource();</p>
<p>然后看lib/CloudForecast/Gearman/Worker.pm里的load_resource(),其实就是根据具体监控项require并且new一个CloudFarecast::Data::*(这个new方法是通过use base和SUPER::new最终到的CloudFarecast.pm上的)。</p>
<p>然后看fetcher_worker()的下一句”$resource->exec_fetch;”,先去找CloudFarecast::Data::*,发现没有exec_fetch(),那往base的CloudFarecast::Data上看,果然有了。其中的主要两行”$ret = $self->do_fetch();”和”$self->call_updater($ret);”。</p>
<p>然后看do_fetch()。其中主要两行”my $ret = $self->fetcher_func->($self);”和”my $schema = $self->rrd_schema;”。这两个fetcher_func和rrd_schema都是前面mk_classdata出来的方法。而这里的$self,则一直追溯到最前面Worker.pm里的$resource,即CloudForecast::Data::*。</p>
<p>选一个CloudForecast::Data::Basic看,其中分别调用了Data.pm里的rrds/graph/title/fetcher函数。</p>
<p>返回Data.pm看fetcher()函数如下:<br />
<code class="highlighter-rouge">perlsub fetcher(&) {
my $class = caller;
Carp::croak("already seted fetcher_func") if $class->fetcher_func;
$class->fetcher_func(shift);
}</code><br />
学习一下,这里新出现的一个caller函数,这是perl自带的函数,可以使用perldoc -f caller查看详细说明。默认返回三个值,分别是调用的package/file/linenumber。显然这里就是获取package,也就是$class = ‘CloudFarecast::Data::Basic’了。然后返回的”$class->fetcher_func(shift);”,这个shift也就是(&)里的内容,即Basic.pm里的{my $c = shift;my @map = …;my $ret = $c->component(‘SNMP’)->get(@map);return $ret;}这个匿名函数。<br />
这样前面Worker里的$resource就有了自己的fetcher_func函数了,就此执行并且返回$ret。完成!</p>
<p>然后把$ret传递给call_updater()函数。这个函数中先对配置文件做一次判断,是否enable了gearmand。如果没有,直接调用exec_updater()完成本地rrd图像的初始化init_rrd()或更新update_rrd()。如果有,则连接上gearmand,new一个CloudForecast::Gearman对象,使用updater方法提交数据。</p>
<p>然后看Gearman.pm中的updater(),其实就是Gearman::Client的dispatch_background()连接上updater任务,发送数据。</p>
cdn自主监控(六):数据展示页面
2011-08-02T00:00:00+08:00
monitor
html
http://chenlinux.com/2011/08/02/calendar-select-html-for-self-monitor
<p>接下来进入我不擅长的页面部分了。规划页面分为side和main,side中提供时间选择框/类型下拉选择框和提交按钮;main中展示最后形成的chart。<br />
时间选择框是用js和css完成的,这个网上有很多,不过要同时支持多浏览器和分钟级别的选项的,目前就发现一个好用的。下载地址如下:<a href="http://chenlinux.com/images/uploads/Calendar.zip">http://chenlinux.com/images/uploads/Calendar.zip</a><br />
然后创建cachemoni/public/cdn.html如下:<br />
```html</p>
<html>
<head>
<meta name="ROBOTS" content="NOINDEX, NOFOLLOW" />
<title>CDN Monitor</title>
</head>
<frameset border="0" frameborder="0" framespacing="0" cols="270,*">
<frame src="side.html" name="side" target="main">
<frame src="main.html" name="main">
```
cachemoni/public/side.html如下:
```html
<link type="text/css" rel="stylesheet" href="css/calendar.css" />
<script language="javascript" src="javascripts/calendar.js"></script>
select_time<hr />
<form action="/cdncharts" method="get" target="main">
<li>begin</li>
<input name="timefrom" type="text" id="timefrom" style="width:100%;" onclick="displayCalendar(this, 'yyyy-mm-dd hh:ii', this, true, '');" />
<li>end</li>
<input name="timeto" type="text" id="timeto" style="width:100%;" onclick="displayCalendar(this, 'yyyy-mm-dd hh:ii', this, true, '');" /><hr />
<select name="chartstype" id="chartstype">
<option value="area">area</option>
<option value="isp">isp</option>
<option value="time" selected="">time</option>
</select>
<input type="submit" value="submit" id="submit" />
</form>```
cachemoni/views/charts.tt如下:
```html
<html>
<head>
<title>FusionCharts Free Documentation</title>
<link rel="stylesheet" href="css/chartstyle.css" type="text/css" />
<script language="JavaScript" src="javascripts/FusionCharts.js"></script>
</head>
<body>
<table width="98%" border="0" cellspacing="0" cellpadding="3" align="center">
<tr>
<td valign="top" class="text" align="center"> <div id="chartdiv" align="center">
FusionCharts.
<script type="text/javascript">
var chart = new FusionCharts("[% IF line %]chartswf/FCF_MSLine.swf[% ELSE %]chartswf/FCF_MSBar2D.swf[% END %]", "ChartId", "400", "350");
chart.setDataURL(escape("[% url %]"));
chart.render("chartdiv");
</script>
```
cachemoni/lib/cachemoni.pm中相关函数如下:
```perl
use Time::Local;
get '/cdncharts' => sub {
my $type = params->{chartstype};
my $begin = unix_time_format(params->{timefrom});
my $end = unix_time_format(params->{timeto});
my $req_url = "/xml?begin=${begin}&end=${end}&type=${type}";
my $line = 1 if $type eq 'time';
template 'charts', { line => $line, url => "$req_url", };
};
sub unix_time_format {
my $time = shift;
if ( $time =~ m/^(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2})/ ) {
return timelocal('00',$5,$4,$3,$2-1,$1-1900);
};
};```
这里比较怪的是,如果setDataURL里传的arg是直接2011-08-01 11:00的格式,fusionchart.js不会发起请求,只有1311111111才行,所以只能在用Time::Local模块转换时间了。
最终访问结果如下:
<img src="/images/uploads/calendar.png" alt="" title="QQ截图20110802191306" width="697" height="370" class="alignnone size-full wp-image-2555" />
</div></td></tr></table></body></html></frame></frame></frameset></html>
cdn自主监控(五):生成charts图像
2011-08-01T00:00:00+08:00
monitor
funsioncharts
http://chenlinux.com/2011/08/01/fusioncharts-for-self-monitor
<p>上篇完成了xml的输出,这篇开始说charts图像的生成。我们采用fusioncharts的免费版,之前博客有提到另一个amcharts,不过amcharts的免费版会在图像左上角加上自己amcharts.com的广告标识……<br />
从fusioncharts官网下载free版的压缩包,有20MB大,不过其中对我们这个小项目有用的只有MSLine.swf/MSBar2D.swf/fusioncharts.js等。<br />
说明:<br />
1、官网介绍上写了支持perl,不过下载包里只有php/asp/jsp/rails的class,没有perl的。所以还是得采用js的方式;<br />
2、js下有setDataURL和setDataXML两个方法,不过官方强烈建议使用setDataURL方法,这种方法可接受的数据量比setDataXML大很多。<br />
页面html很简单,加如下js代码即可:<br />
<code class="highlighter-rouge">javascript</script>
<script type="text/javascript">
var myChart = new FusionCharts("/Charts/MSBar2D.swf", "myChartId", "600", "500");
myChart.setDataURL(escape("/xml?begin=***&end=***&type=area"));
myChart.render("chartdiv");
</script></code><br />
要点在这个escape(),如果URL里带参数的话,必须用escape()转码,不然的话第一个&之后的所有参数都会丢失掉!<br />
在调试中注意到,为了忽略缓存,setDataURL()会在url最后跟上一个随机1-4位数字参数&curr=1234然后再发起请求。<br />
现在就可以访问页面看看了~嗯,可以看到图像中的中文字有问题。这是因为fusioncharts不认utf8的中文,必须输出gbk或者gb2312的xml数据才行。所以需要修改一些dancer的charset配置,把config.yml里的charset: UTF-8改成charset: GBK。然后重新请求,中文就正确显示了。<br />
如下图:<br />
<img src="/images/uploads/fusioncharts.jpg" alt="" title="MSBar2D" width="400" height="350" class="alignnone size-full wp-image-2548" /></p>
<hr />
<p>题外话:因为数据是自己insert的,结果写了个区号0751,在quhao.txt里不存在,导致输出category的时候总有问题……难怪总看人说程序本身很好写,各种错误处理才是麻烦事……</p>
cdn自主监控(四):输出xml数据
2011-07-28T00:00:00+08:00
monitor
dancer
http://chenlinux.com/2011/07/28/output-xml-for-funsioncharts
<p>准备使用funsioncharts绘图,其采用xml数据,在绘制line图的时候,就要从mysql里读取数据,并输出成xml格式,相关配置如下:<br />
```perl<br />
package cachemoni;<br />
use Dancer ‘:syntax’;<br />
use Dancer::Plugin::Database;<br />
use POSIX qw(strftime);</p>
<p>get ‘/xml’ => sub {<br />
my $begin_time = date_format(params->{begin});<br />
my $end_time = date_format(params->{end});<br />
my $type = params->{type};<br />
my $color = { chinacache => ‘1D8BD1’,<br />
dnion => ‘F1683C’,<br />
fastweb => ‘2AD62A’,<br />
};<br />
my $xml_head = “<graph caption="Response Time" subcaption="from $begin_time to $end_time" hovercapbg="FFECAA" hovercapborder="F47E00" formatnumberscale="0" decimalprecision="0" showvalues="0" numdivlines="3" numvdivlines="0" yaxisminvalue="1000" yaxismaxvalue="1800" rotatenames="1">\n<categories>\n";
my $group;
if ( $type eq 'time' ) {
$group = 'cur_date';
} elsif ( $type eq 'isp' ) {
$group = 'isp';
} elsif ( $type eq 'area' ) {
$group = 'area';
} else {
return 'Error';
};
my $xml = cdn_select($begin_time, $end_time, $group, $color, $xml_head);
return $xml;
};</categories></graph></p>
<p>sub get_area_code {<br />
my $file = shift;<br />
my $area_code = { ‘0000’ => ‘其他’ };<br />
open my $fh,’<’,”$file” or die “Cannot open $file”;<br />
while (<$fh>) {<br />
chomp;<br />
my($area,$code) = split;<br />
$area_code->{“$code”} = “$area”;<br />
}<br />
close $fh;<br />
return $area_code;<br />
};</p>
<p>sub cdn_select {<br />
my ($begin_time, $end_time, $group, $color, $xml) = @_;<br />
my $sql = “SELECT ${group},AVG(avg_time) avg FROM cdn_cron_record WHERE cdn = ? AND cur_date BETWEEN ? AND ? GROUP BY ${group} ORDER BY ${group}”;<br />
my $sth = database->prepare($sql);<br />
my $i = 0;<br />
for my $cdn (qw{chinacache dnion fastweb}) {<br />
$sth->execute($cdn, $begin_time, $end_time);<br />
unless($i) {<br />
my @values;<br />
while ( my $ref = $sth->fetchrow_hashref ) {<br />
my ($avg_time, $type) = ($ref->{‘avg’}, $ref->{“$group”});<br />
$xml .= “<category name="convert_group($group, $type)"></category>\n”;<br />
push @values, $avg_time;<br />
};<br />
$xml .= “</category>\n”;<br />
$xml .= “<dataset seriesname="$cdn" color="$color->{$cdn}">\n";
$xml .= "<set value="$_"></set>\n" for @values;
$xml .= "</dataset>\n”;<br />
} else {<br />
$xml .= “<dataset seriesname="$cdn" color="$color->{$cdn}">\n";
while ( my $ref = $sth->fetchrow_hashref ) {
$xml .= "<set value="$ref->{avg}"></set>\n";
};
$xml .= "</dataset>\n”;<br />
};<br />
$i++;<br />
};<br />
$xml .= ‘</graph>’;<br />
return $xml;<br />
};</p>
<p>sub convert_group {<br />
my ($group, $origin) = @_;<br />
if ($group eq ‘cur_date’) {<br />
return $origin;<br />
} elsif ($group eq ‘isp’) {<br />
my @isplist = qw(其他 电信 联通 移动 教育网);<br />
return $isplist[$origin];<br />
} elsif ($group eq ‘area’) {<br />
my $arealist = get_area_code(‘quhao.txt’);<br />
my $code = sprintf(“%04s”,$origin);<br />
return $arealist->{$code};<br />
} else {<br />
return ‘Error’;<br />
};<br />
};</p>
<p>sub date_format {<br />
my $time = shift;<br />
return strftime(“%F %H:%M”,localtime($time)) if $time =~ m/\d+/;<br />
};<code class="highlighter-rouge">
用curl请求一下,如下:
</code>bash[root@naigos ~]# curl ‘http://cache.monitor.china.com/xml?begin=1311739200&end=1311840000&type=time’</p>
<graph caption="Daily Visits" subcaption="from 2011-07-27 12:00 to 2011-07-28 16:00" hovercapbg="FFECAA" hovercapborder="F47E00" formatnumberscale="0" decimalprecision="0" showvalues="0" numdivlines="3" numvdivlines="0" yaxisminvalue="1000" yaxismaxvalue="1800" rotatenames="1"><categories><category name="2011-07-27 17:27:12" />
<category name="2011-07-27 17:32:12" />
<category name="2011-07-27 17:37:01" />
<dataset seriesname="chinacache" color="1D8BD1"><set value="300.0000" />
<set value="511.0000" />
<set value="482.0000" />
</dataset><dataset seriesname="dnion" color="F1683C"><set value="595.6667" />
<set value="432.0000" />
</dataset><dataset seriesname="fastweb" color="2AD62A"><set value="471.0000" />
<set value="431.5000" />
</dataset>```
换个area参数试试:
```bash[root@naigos lib]# curl 'http://cache.monitor.china.com/xml?begin=1311739200&end=1312169668&type=area'
<graph caption="Response Time" subcaption="from 2011-07-27 12:00 to 2011-08-01 11:34" hovercapbg="FFECAA" hovercapborder="F47E00" formatnumberscale="0" decimalprecision="0" showvalues="0" numdivlines="3" numvdivlines="0" yaxisminvalue="1000" yaxismaxvalue="1800" rotatenames="1">
<categories>
<category name="四川" />
<category name="江西" />
<dataset seriesname="chinacache" color="1D8BD1">
<set value="511.0000" />
<set value="421.3333" />
</dataset><dataset seriesname="dnion" color="F1683C">
<set value="482.5000" />
</dataset><dataset seriesname="fastweb" color="2AD62A">
<set value="431.3333" />
</dataset>```
呃,上面用的数据只是我自己insert的,所以出现了比较囧的条数不一致……
</categories></graph></categories></graph>
cdn自主监控(三):数据库准备工作
2011-07-27T00:00:00+08:00
monitor
MySQL
http://chenlinux.com/2011/07/27/prepare-mysql-for-self-monitor
<p>准备两个表,一个存储原始数据,另一个存储每5分钟归总一次的数据。之后根据时间段绘制省份运营商性能图的时候,就直接从汇总表里获取数据;原始表留给详细查询。<br />
数据库准备脚本如下:<br />
```mysqlUSE myops;<br />
CREATE TABLE IF NOT EXISTS cdn_ori_record (<br />
id INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY,<br />
ip INT(10) NOT NULL DEFAULT ‘0000000000’,<br />
isp ENUM(‘0’,’1’,’2’,’3’,’4’),<br />
area INT(4) NOT NULL DEFAULT ‘0000’,<br />
cur_date TIMESTAMP DEFAULT NOW(),<br />
cdn_time INT(10) NOT NULL DEFAULT ‘0’,<br />
cdn ENUM(‘CHINACACHE’,’DNION’,’FASTWEB’) NOT NULL,<br />
KEY time_key (cur_date)<br />
);</p>
<p>CREATE TABLE IF NOT EXISTS cdn_cron_record (<br />
id INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY,<br />
isp TINYINT(1) NOT NULL DEFAULT ‘0’,<br />
area INT(4) NOT NULL DEFAULT ‘0000’,<br />
cur_date TIMESTAMP DEFAULT NOW(),<br />
cdn ENUM(‘CHINACACHE’,’DNION’,’FASTWEB’) NOT NULL,<br />
avg_time INT(10) NOT NULL DEFAULT ‘0’,<br />
KEY time_area_isp_key (cur_date, area, isp)<br />
);</p>
<p>DELIMITER |<br />
DROP PROCEDURE IF EXISTS cdn_cron |<br />
CREATE PROCEDURE cdn_cron()<br />
BEGIN<br />
INSERT INTO cdn_cron_record(isp,area,cdn,avg_time) <br />
SELECT isp,area,cdn,AVG(cdn_time) FROM cdn_ori_record <br />
WHERE cur_date > FROM_UNIXTIME(UNIX_TIMESTAMP()-300)<br />
GROUP BY cdn,area,isp;<br />
END |<br />
DELIMITER ;</p>
<p>SET GLOBAL event_scheduler = 1;<br />
CREATE EVENT IF NOT EXISTS event_cdn ON SCHEDULE EVERY 300 SECOND ON COMPLETION PRESERVE DO CALL cdn_cron();<br />
ALTER EVENT event_cdn ON COMPLETION PRESERVE ENABLE;<br />
```</p>
cdn自主监控(二):快速查找ip对应信息
2011-07-26T00:00:00+08:00
monitor
perl
http://chenlinux.com/2011/07/26/seek-iplist-for-self-monitor
<p>话接上篇,ip整理出来,然后就是接一个ip地址,快速定位查找到它属于哪个ip段,然后返回具体的省份和运营商。因为之前的ip已经转换成数字,而且是顺序排列的,所以可以采用折半算法(二分法)。perl脚本如下:<br />
```perl#!/usr/bin/perl -w<br />
#my $ip = inet_aton(“$ARGV[0]”);<br />
my $ip = inet_aton(get_test_ip());<br />
my $file = $ARGV[1] || ‘iplist.txt’;<br />
my $length = <code class="highlighter-rouge">cat $file | wc -l</code>;</p>
<p>my $code = get_area_code(‘quhao.txt’);<br />
my @isplist = qw(‘其他’ ‘电信’ ‘联通’ ‘移动’ ‘教育网’);</p>
<p>open my $fh, ‘<’, “$file” or die “Cannot open $file\n”;<br />
my $line_len = ‘26’; #=10+10+4+1+1,包括回车符(注意上篇输出为了好看多了空格,可以删掉) <br />
my $first = 0;<br />
my $last = $length - 1; #统一使用SEEK_SET,所以最后一行的起始位置是length-1 <br />
my $result = 1;<br />
while ($result) {<br />
my $middle = sprintf(“%.0f”,($last-$first) / 2 + $first); #折半位置,除法取整时采用sprintf比直接int精确<br />
seek $fh, $line_len * $middle, 0; #移动到折半位置 <br />
sysread $fh, $begin_ip, 10; #从折半处读取10位 <br />
sysread $fh, $end_ip, 10; #接着再读10位,如果没删空格,还要先seek移动1位,麻烦 <br />
#根据比大小决定下次向哪个方向折半<br />
if ( $ip < $begin_ip ) {<br />
$last = $middle;<br />
next;<br />
} elsif ( $ip > $end_ip ) {<br />
$first = $middle;<br />
next;<br />
} else {<br />
#找到相应区间,读取区号和运营商号<br />
sysread $fh, $area, 4;<br />
sysread $fh, $isp, 1;<br />
printf “%010s %s %s\n”, $ip, $code->{“$area”}, $isplist[$isp];<br />
$result = 0; #设定$result为假,退出循环 <br />
};<br />
};</p>
<p>close $fh;</p>
<p>sub inet_aton {<br />
my $ip = shift;<br />
my $short = sprintf “%010s”, $1 * 256<strong>3 + $2 * 256</strong>2 + $3 * 256 + $4 if $ip =~ /(\d+).(\d+).(\d+).(\d+)/;<br />
return $short;<br />
};<br />
#对应区号和省份,跟上篇的kv相反<br />
sub get_area_code {<br />
my $file = shift;<br />
my $area_code = { ‘0000’ => ‘other’ };<br />
open my $fh,’<’,”$file” or die “Cannot open $file”;<br />
while (<$fh>) {<br />
chomp;<br />
my($area,$code) = split;<br />
$area_code->{“$code”} = “$area”;<br />
}<br />
close $fh;<br />
return $area_code;<br />
};<br />
#生成一个随机的合法ip地址<br />
sub get_test_ip {<br />
return join ‘.’, map int rand 256, 1..4;<br />
}```</p>
cdn自主监控(一):整理一个可用范围内的尽可能小的ip库
2011-07-25T00:00:00+08:00
monitor
perl
http://chenlinux.com/2011/07/25/overwrite-qqwry-for-self-monitor
<p>为了跟第三方监控做对比和作为备用,准备自己通过页面js返回数据做个监控。首先第一步,整理一个足够自己用的ip库。<br />
首先,考虑调用会比较频繁,打算把内容尽可能归并,到省级运营商即可;<br />
其次,未知归并完会有多大的情况下,考虑到qqwry的地区都是中文,打算统一使用电话区号代替地区,运营商也有一位数字代替,ip采用inet-aton网络值代替;这样每条记录的字节数固定,可以方便采用seek和sysread提高读取某条记录的速度。</p>
<hr />
<p>前几步工作和之前整理ip库给dns用的时候一样,导出qqwry.txt,约42w条,24MB大小。<br />
然后从网上搜一下全国电话区号,以各省省会(首府)的区号为准,存成一个quhao.txt,为了之后处理方便,只保留前两个中文,好在中国的省份里也只有内蒙古和黑龙江是三个字,留两位不至于影响阅读,txt内容如下:<br />
<code class="highlighter-rouge">yaml北京 0010
上海 0021
天津 0022
重庆 0023
安徽 0551
福建 0591
甘肃 0931
广东 0020
广西 0771
贵州 0851
海南 0898
河北 0311
河南 0371
黑龙 0451
湖北 0027
湖南 0731
吉林 0431
江苏 0025
江西 0791
辽宁 0024
内蒙 0471
宁夏 0951
青海 0971
山东 0531
山西 0351
陕西 0029
四川 0028
西藏 0891
新疆 0991
云南 0871
浙江 0571</code><br />
然后用如下perl脚本归并:<br />
```perl#!/usr/bin/perl<br />
use strict;<br />
use warnings;<br />
my($quhao, $qqwry) = @ARGV;<br />
$code = read_area_code($quhao);<br />
overwrite_iplist($qqwry, $code);<br />
sub inet_aton {<br />
my $ip = shift;<br />
my $short = sprintf “%010s”, $1 * 256<strong>3 + $2 * 256</strong>2 + $3 * 256 + $4 if $ip =~ /(\d+).(\d+).(\d+).(\d+)/;<br />
return $short;<br />
};</p>
<p>sub read_area_code {<br />
my $file = shift;<br />
my $area_code = {};<br />
open my $fh,’<’,”$file” or die “Cannot open $file”;<br />
while (<$fh>) {<br />
chomp;<br />
my($area,$code) = split;<br />
$area_code->{“$area”} = “$code”;<br />
}<br />
close $fh;<br />
return $area_code;<br />
};</p>
<p>sub overwrite_iplist {<br />
my($iplist, $area_code) = @_;<br />
my($last_begin_ip_n, $last_end_ip_n, $last_province_n, $last_isp_n);<br />
open my $fh,’<’,”$iplist” or die “Cannoet open $iplist”;<br />
while (<$fh>) {<br />
chomp;<br />
my($begin_ip, $end_ip, $area, $isp) = split;<br />
my($province_n, $isp_n);<br />
my $begin_ip_n = &inet_aton(“$begin_ip”);<br />
my $end_ip_n = &inet_aton(“$end_ip”);<br />
next if ($end_ip_n - $begin_ip_n) < 32;<br />
if ( $area =~ /学/ ) {<br />
$isp_n = 4; #教育网, 因为清华北大等学校记录在area里了,所以这步提前设定 <br />
};<br />
if ( $isp =~ m/电信/ ) {<br />
$isp_n = 1; #电信 <br />
} elsif ( $isp =~ m/联通/ ) {<br />
$isp_n = 2; #联通(包括原网通) <br />
} elsif ( $isp =~ m/铁通|移动/ ) {<br />
$isp_n = 3; #移动(包括原铁通) <br />
} elsif ( $isp =~ m/学/ ) {<br />
$isp_n = 4; #教育网 <br />
} else {<br />
$isp_n = 0; #国外地址及其他未能识别的国内运营商 <br />
};</p>
<div class="highlighter-rouge"><pre class="highlight"><code> my $province = substr($area, 0, 4); #中文用2字节,所以对原始记录获取前四字节即为省份名
if ( exists $area_code->{"$province"} ) {
$province_n = $area_code->{"$province"}; #国内已知电话区号的省份
} else {
$province_n = '0000'; #港澳台及外国,可能有其他未能识别的国内地址
};
#下段为合并网段,之前dns时也用过
if (!$last_province_n) {
($last_begin_ip_n, $last_end_ip_n, $last_province_n, $last_isp_n) = ($begin_ip_n, $end_ip_n, $province_n, $isp_n);
};
if ( $last_province_n == $province_n && $last_isp_n == $isp_n ) {
$last_end_ip_n = $end_ip_n;
} else {
printf "%010s %010s %04s %s\n", $last_begin_ip_n, $last_end_ip_n, $last_province_n, $last_isp_n;
($last_begin_ip_n, $last_end_ip_n, $last_province_n, $last_isp_n) = ($begin_ip_n, $end_ip_n, $province_n, $isp_n);
};
};
close $fh; }; ``` 最后运行如下命令即可获得精简ip库: ```bashperl overwrite.pl quhao.txt qqwry.txt > newip.txt``` 对比一下大小和行数: ```bash[root@cdn2 ~]# ll -rw-r--r-- 1 root root 539226 Jul 25 18:18 newer.txt -rw-r--r-- 1 root root 9058928 May 15 13:13 QQWry.dat -rw-r--r-- 1 root root 24753935 Jul 25 15:26 qqwry.txt -rw-r--r-- 1 root root 340 Jul 25 15:03 quhao.txt -rw-r--r-- 1 root root 2439 Jul 25 18:17 test.pl [root@cdn2 ~]# wc -l * 18594 newer.txt 34727 QQWry.dat 428454 qqwry.txt
30 quhao.txt
72 test.pl``` 好了,一天一天来,明天实现在这527KB的文件里快速定位ip……
</code></pre>
</div>
perl小测试题(转自CU)
2011-07-21T00:00:00+08:00
perl
http://chenlinux.com/2011/07/21/some-tests-about-perl
<p>原贴地址:http://bbs.chinaunix.net/thread-3563215-1-1.html<br />
flw回帖里有翻译,我试答如下:<br />
1. Perl 5 中变量名开头的那个字符(sigils)有哪几种、分别都有什么含义?<br />
$标量@数组%散列&函数<br />
2. 访问一个数组元素时,用 $items[$index] 和用 @items[$index] 有什么区别?<br />
$是标量,@是数组切片<br />
3. 请问 == 和 eq 之间的区别是什么?<br />
分别是数值和字符的比较<br />
4. 在列表上下文中对一个 hash 求值,会得到什么?<br />
应该是得到一个数组?<br />
5. 如何查看关键字的 Perl 文档?<br />
perldoc -f <br />
6. Perl 5 中的函数和方法有什么不同?<br />
方法是bless后的函数调用?<br />
7. Perl 5 什么时候对一个变量所使用的内存进行回收?<br />
引用计数器清空后?(好像记得是)或者执行完成退出的时候<br />
8. 如何做才能够确保一个变量的缺省作用域是词法作用域?<br />
my local?<br />
9. 如何加载模块并且从中导入符号?<br />
use Module;<br />
符号是啥?<br />
10. 是什么在控制 Perl 如何加载模块?有什么办法可以指定一个目录清单告诉 Perl 从这些地方尝试加载模块?<br />
不知道。<br />
use lib qw();或者perl -I或者export PERL5LIB=<br />
11. 你怎么在 Perl 5 的文档中中查看一条错误信息说明?(加分项:了解如何为所有遇到过的错误信息启用解释)<br />
汗,这个用|less然后/搜索……有好办法么?<br />
12. 试着阐述一下把数组传递给函数的时候会发生什么事。<br />
应该是复制一个数组副本出来,然后赋值到@<em>给函数用?<br />
13. 如何传递多个独立的数组给函数?<br />
分别传递数组引用。<br />
14. 对于调用方来说,return 和 return undef 有什么区别?<br />
return的应该是上一个的结果吧?<br />
15. Where do tests go in a standard CPAN distribution?<br />
module里头都有t/目录可以test吧,然后cpan.org上有志愿者?<br />
16. 拿到一个 CPAN 模块后怎样进行测试?<br />
make test<br />
17. 你用什么命令从 CPAN 上安装新模块?<br />
cpanm Module<br />
18. 为什么要使用 open 函数的三参数形式?<br />
预防文件名带有>等特殊字符串<br />
19. 如何检测(和报告)像 open 这样的系统调用产生的错误?(加分项: 知道如何开启自动检测和报告)<br />
在open的时候接上or die $@;<br />
use autodie;<br />
20. 如何在 Perl 5 中抛出一个异常?<br />
die<br />
21. 如何在 Perl 5 中捕获一个异常?<br />
eval {}<br />
22. 用 for 读文件和用 while 读文件有什么不同吗?<br />
for一次性读入;while一次一行<br />
23. 在 Perl 5 的函数或者方法中,你分别是如何处理参数的?<br />
my $abc = shift;<br />
my ($abc, $def) = @</em>;<br />
24. my ($value) = @_; 中的小括号有什么用?如果删掉会发生什么?<br />
强制为列表环境;<br />
删掉之后变成标量环境就是@_的元素个数了。<br />
25. new 是 Perl 5 的内置函数或者关键字吗?<br />
不是,模块里要自己写new方法。<br />
26. 你怎么看 Perl 的核心模块的文档?如果是 CPAN 模块的话又该如何看?<br />
perldoc查看,不过核心模块和一般模块方法有区别么?<br />
27. 怎样只访问 hash 的值?<br />
values函数</p>
mysql_history_monitor
2011-07-08T00:00:00+08:00
monitor
MySQL
http://chenlinux.com/2011/07/08/mysql_history_monitor
<p>上篇加了bash_history的监控,这篇说mysql_history的监控。不像bash4,mysql自始至终没有提供过syslog的代码,只能自己通过守护进程去实时获取~/.mysql_history的记录了。一个小脚本如下:<br />
```perl#!/usr/bin/perl -w<br />
use POE qw(Wheel::FollowTail);<br />
use Log::Syslog::Fast qw(:all);</p>
<p>defined(my $pid = fork) or die “Cant fork:$!”;<br />
unless($pid){ <br />
}else{<br />
exit 0;<br />
}</p>
<p>POE::Session->create(<br />
inline_states => {<br />
<em>start => sub {<br />
$</em>[HEAP]{tailor} = POE::Wheel::FollowTail->new(<br />
Filename => “/root/.mysql_history”,<br />
InputEvent => “got_log_line”,<br />
ResetEvent => “got_log_rollover”,<br />
);<br />
},<br />
got_log_line => sub {<br />
#通过Data::Dumper看到实际是$<em>[10],不过在POE::Session里定义了sub ARG0 () { 10 };这样写起来简单了<br />
to_rsyslog($</em>[ARG0]);<br />
},<br />
got_log_rollover => sub {<br />
to_rsyslog(‘roll’);<br />
},<br />
}<br />
);</p>
<p>POE::Kernel->run();<br />
exit;</p>
<p>sub to_rsyslog {<br />
$message = join’ ‘,@_;<br />
#rsyslog开的是UDP的514端口;而LOG_LOCAL0和LOG_INFO都是syslog定义的,乱写的话会自动归入kernel | alert<br />
my $logger = Log::Syslog::Fast->new(LOG_UDP, “10.0.0.123”, 514, LOG_LOCAL0, LOG_INFO, “mysql_231”, “mysql_monitor”);<br />
$logger->send($message ,time);<br />
};```</p>
<p>当然,mysql的history其实不止一个位置,需要判断~</p>
bash_syslog_history
2011-07-08T00:00:00+08:00
monitor
bash
http://chenlinux.com/2011/07/08/bash_syslog_history
<p>@钚钉钬影 童鞋要试验系统日志收集处理,采用rsyslog+loganalyzer做界面,因为需求有把bash_history也带上,所以采用修改bash源码的方式。最先采用了bash4.2(bash4.2已经带了这个功能,但是默认不开启,删掉哪个/**/就行了),不过因为这个bash4.+跟redhat的network-functions脚本有点小矛盾,重启的时候报个错——虽然改起来也很简单,就是加一个./的事情——不过本着尽量改动少的原则,换回bash3.1。<br />
bash3.1里默认没有记录syslog的代码,所以从bash4.2/bashhist.c里复制有关bash_syslog_history代码段到bash3.1中——注意不是原样复制到bashhist.c,那样不顶用——需要复制到lib/readline/history.c中,如下:<br />
```c<br />
+ #include <syslog.h>
……
void
add_history (string)
const char *string;
{
HIST_ENTRY *temp;</syslog.h></p>
<p>+if (strlen(string)<600) {<br />
+ syslog(LOG_LOCAL5 | LOG_INFO, “%s %s %s %s”,getenv(“LOGNAME”),getlogin(),ttyname(0),string);<br />
+ }<br />
+ else {<br />
+ char trunc[600];<br />
+ strncpy(trunc,string,sizeof(trunc));<br />
+ trunc[sizeof(trunc)-1]=’\0’;<br />
+ syslog(LOG_LOCAL5, LOG_INFO, “%s %s %s %s(++TRUNC)”,getenv(“LOGNAME”),getlogin(),ttyname(0), trunc);<br />
+ }</p>
<p>if (history_stifled && (history_length == history_max_entries))<br />
```<br />
上面的LOG_LOCAL5和LOG_INFO都是syslog.h里的定义,所以要include进来。<br />
然后编译使用,就能记录进/var/log/messages里了。</p>
<hr />
<p>话说网上关于上面的说明中,大多数没有提到需要include<syslog.h>,但是make的时候居然只报LOG_INFO和LOG_LOCAL5的未定义错误,随便把这两个改成一个已经defined过的变量,比如HISTORY,编译一样成功;而且也一样记录系统日志。唯一体现不同的地方就是loganalyzer页面上看到所有的history操作都是alert级别。
之前没了解过程的时候,还采用了mysql触发器,自动修改这个报警级别,也记录一下,毕竟是自己第一次用触发器~~
```mysqlUSE Syslog;
DROP TRIGGER IF EXISTS trig_bash_prior;
DELIMITER |
CREATE TRIGGER trig_bash_prior BEFORE INSERT ON Syslog.SystemEvents
FOR EACH ROW BEGIN
IF NEW.SysLogTag='-bash:' && NEW.Priority='1' THEN
SET NEW.Priority='6';
END IF;
END
|
```
这个库很简单,基本数据就是记在这个单表里~~</syslog.h></p>
squid监控+dancer小试验
2011-06-28T00:00:00+08:00
dancer
squid
perl
http://chenlinux.com/2011/06/28/a-dancer-demo-of-monitor-squid
<p>squid监控之前有一篇关于snmp的内容,不过这次是真要用上了,所以细细挑出来几个做监控。碰巧凯哥更新了一把modern perl的东西,我亦步亦趋,也试试dancer。不过花了两天时间,DBIx::Class::Schema还是没搞出来,最终还是简单的用DBI跳过了……<br />
用的database就是之前nmap试验时生成的数据,有application/channel/intranet等column。<br />
首先安装:<br />
<code class="highlighter-rouge">perlcpanm Dancer DBI DBD:mysql Template Dancer::Session::YAML
dancer -a cachemoni
</code><br />
然后修改cachemoni/lib/cachemoni.pm如下:<br />
```perl<br />
package cachemoni;<br />
use Dancer ‘:syntax’;<br />
use Dancer::Plugin::Database;<br />
use Net::SNMP;<br />
use Digest::MD5 qw(md5_hex);<br />
our $VERSION = ‘0.1’;</p>
<p>get ‘/monitor’ => sub {<br />
my $app = params->{app} || ‘squid’;<br />
return ‘Only support squid now’ unless $app eq ‘squid’;<br />
my $checkstat = _snmp_check($app);<br />
template ‘monitor’, { status => $checkstat,<br />
name => [‘IP地址’,’流量命中率’,’请求数命中率’,’回源请求响应毫秒’,’当前客户端数’,’剩余文件描述符数’,’已缓存文件数’,’平均每秒请求数’],<br />
};<br />
};</p>
<p>any[‘get’, ‘post’] => ‘/login’ => sub {<br />
my $err;<br />
if ( request->method() eq ‘POST’ ) {<br />
my $name = params->{username};<br />
my $passwd = params->{password};<br />
my $sth = database->prepare(<br />
‘select audit,passwd from ops_user where name = ?’<br />
);<br />
$sth->execute($name);<br />
my $ref = $sth->fetchrow_arrayref;<br />
if (!defined $ref) {<br />
redirect ‘/register’;<br />
} elsif ( $ref->[0] ne ‘yes’ ) {<br />
$err = ‘你的帐户还没通过审核’;<br />
} elsif ( md5_hex(“$passwd”) ne “$ref->[1]” ) {<br />
$err = ‘密码错误’;<br />
} else {<br />
session ‘logged_in’ => $name;<br />
redirect ‘/’;<br />
};<br />
};<br />
template ‘login’, { ‘err’ => $err, };<br />
};</p>
<p>any[‘get’, ‘post’] => ‘/register’ => sub {<br />
my $err;<br />
if ( request->method() eq ‘POST’ ) {<br />
my $name = params->{username};<br />
my $passwd = params->{password};<br />
my $check_sth = database->prepare(<br />
‘select count(name) from ops_user where name = ?’<br />
);<br />
$check_sth->execute($name);<br />
my $ref = $check_sth->fetchrow_arrayref;<br />
if ( “$ref->[0]” == ‘0’ ) {<br />
my $insert_sth = database->prepare(<br />
‘insert into ops_user (name,passwd) value(?,?)’<br />
);<br />
$insert_sth->execute($name, md5_hex($passwd));<br />
$err = ‘等待人工审核通过,3Q’;<br />
} else {<br />
$err = ‘该用户名已注册’;<br />
};<br />
};<br />
template ‘register’, { ‘err’ => $err, };<br />
};</p>
<p>get ‘/’ => sub {<br />
template ‘index’;<br />
};</p>
<p>get ‘/logout’ => sub {<br />
session->destroy;<br />
redirect ‘/’;<br />
};</p>
<p>sub _snmp_check {<br />
my $app = shift;<br />
my $list = {};<br />
#之前通过use加载了plugin::database,所以直接有database对象引用了<br />
my $sth = database->prepare(<br />
‘select channel,intranet from myhost where application = ?’,<br />
);<br />
$sth->execute($app);<br />
while (my $ref = $sth->fetchrow_hashref()) {<br />
$list->{“$ref->{‘channel’}”} = [] unless exists $list->{“$ref->{‘channel’}”};<br />
my $ip = $ref->{‘intranet’};<br />
my @snmpstat = _snmp_walk(“$ip”);<br />
#这里第一是没想到比较好的把每个ip的各项检查结果导出的办法;所以干脆采用固定次序输出;<br />
#第二是因为unshift很耗资源,所以先push再reverse<br />
push @snmpstat, $ip;<br />
@snmpstat = reverse @snmpstat;<br />
push @{$list->{“$ref->{‘channel’}”}}, \@snmpstat;<br />
}</p>
<div class="highlighter-rouge"><pre class="highlight"><code>push my @checkstat, map{
{ channel => $_,
host => $list->{"$_"},
}
} keys %{$list};
return \@checkstat; };
</code></pre>
</div>
<p>sub _snmp_walk {<br />
my $o_host = shift;<br />
my $o_community = ‘public’;<br />
my $result = {};<br />
my %oids = (<br />
‘cacheUptime’ => ‘.1.3.6.1.4.1.3495.1.1.3.0’,<br />
‘cacheProtoClientHttpRequests’ => ‘.1.3.6.1.4.1.3495.1.3.2.1.1.0’,<br />
‘cacheNumObjCount’ => ‘.1.3.6.1.4.1.3495.1.3.1.7.0’,<br />
‘cacheCurrentUnusedFDescrCnt’ => ‘.1.3.6.1.4.1.3495.1.3.1.10.0’,<br />
‘cacheClients’ => ‘.1.3.6.1.4.1.3495.1.3.2.1.15.0’,<br />
‘cacheHttpMissSvcTime’ => ‘.1.3.6.1.4.1.3495.1.3.2.2.1.3.1’,<br />
‘cacheRequestHitRatio’ => ‘.1.3.6.1.4.1.3495.1.3.2.2.1.9.1’,<br />
‘cacheRequestByteRatio’ => ‘.1.3.6.1.4.1.3495.1.3.2.2.1.10.1’,<br />
);</p>
<div class="highlighter-rouge"><pre class="highlight"><code>my ($session, $error) = Net::SNMP->session(
-hostname => $o_host,
-community => $o_community, #默认是5秒超时,如果有没开snmp的,这页面打开时间一下就上去了,所以要调短
-timeout => 1,
);
if (defined $session) {
$session->translate(Net::SNMP->TRANSLATE_NONE);
my $squid_status = $session->get_request( -varbindlist => [$oids{'cacheUptime'}, $oids{'cacheProtoClientHttpRequests'}, $oids{'cacheNumObjCount'}, $oids{'cacheCurrentUnusedFDescrCnt'}, $oids{'cacheClients'}, $oids{'cacheHttpMissSvcTime'}, $oids{'cacheRequestHitRatio'}, $oids{'cacheRequestByteRatio'}, ], );
$session->close;
if (defined $squid_status) { #如果要算当前的rps,那得sleep 1再取一次,对几十台集群监控来说不可取; #姑且用总的平均值做个衡量,或许可以采用二八法则估算一个高峰时间的rps; #注意这个cacheUptime是保留两位ms的,所以算rps的时候要除以100; #上述原因也是一般大家对单机squid监控采用rrd的原因,采用COUNTER32数据源,直接递增显示。
my $request_sum = $squid_status->{"$oids{'cacheProtoClientHttpRequests'}"};
my $uptime = $squid_status->{"$oids{'cacheUptime'}"};
return $request_sum / $uptime * 100, $squid_status->{"$oids{'cacheNumObjCount'}"}, $squid_status->{"$oids{'cacheCurrentUnusedFDescrCnt'}"}, $squid_status->{"$oids{'cacheClients'}"}, $squid_status->{"$oids{'cacheHttpMissSvcTime'}"}, $squid_status->{"$oids{'cacheRequestHitRatio'}"}, $squid_status->{"$oids{'cacheRequestByteRatio'}"};
};
}; };
</code></pre>
</div>
<p>true;<code class="highlighter-rouge">
修改cachemoni/config.yml如下:
</code>yaml<br />
appname: “cachemoni”<br />
layout: “main”<br />
charset: “UTF-8”<br />
#采用TT作为view模板,注意tt默认是[%%],而dancer里是<%%>,所以另外要定义标签<br />
template: “template_toolkit”<br />
engines:<br />
template_toolkit:<br />
encoding: ‘utf8’<br />
start_tag: ‘[%’<br />
end_tag: ‘%]’<br />
#用默认的Simple,即存在内存的一个hash里,实际效果很不稳定,改用yaml,但也有yml文件在关闭浏览器后依旧存在的问题。<br />
session: “YAML”<br />
session_dir: “/tmp/dancer_session_dir”<br />
session_expires: 300<br />
plugins:<br />
Database:<br />
driver: ‘mysql’<br />
database: ‘myops’<br />
host: ‘10.1.1.25’<br />
username: ‘user’<br />
password: ‘password’<br />
connection_check_threshold: 10<br />
dbi_params:<br />
RaiseError: 1<br />
AutoCommit: 1<br />
on_connect_do: [“SET NAMES ‘utf8’”, “SET CHARACTER SET ‘utf8’” ]<br />
log_queries: 1<code class="highlighter-rouge">
创建cachemoni/views/monitor.tt如下:
</code>html[% IF session.logged_in %]<br />
[% FOREACH stat IN status %]</p>
<hr />
<p>channel: [% stat.channel %]</p>
<table width="100%" cellspacing="0" cellpadding="0" border="1">
<tr>
[% FOREACH col IN name %]
<th><center>
[% col %]
</center></th>
[% END %]
</tr>
[% FOREACH host IN stat.host %]
<tr>
[% FOREACH list IN host %]
<td>
[% list %]
</td>
[% END %]
[% END %]
</tr></table>
<p>[% END %]<br />
[% ELSE %]<br />
内部信息,请登陆后查看<br />
[% END %]<code class="highlighter-rouge">
创建cachemoni/views/login.tt如下:
</code>html</p>
<center>
<h2>登陆页面</h2>
[% IF err %]<p class=error><strong>Error:</strong> [% err %][% END %]
<form action="/login" method=post>
<dl>
<dt>用户名:
<input type=text name=username>
<dt>密 码:
<input type=password name=password>
<dd><input type=submit value=login>
```注册页面类似,不贴了。
然后是layout层的共用部分,之前定义了是views/layouts/main.tt,如下:
```html……
<body>
<div class=metanav>
<a href="/">首页</a> |
[% IF not session.logged_in %]
<a href="/register">注册</a> |
<a href="/login">登陆</a>
[% ELSE %]
<a href="/logout">退出</a>
[% END %]
[% content %]
<div id="footer">
Powered by <a href="http://perldancer.org/">Dancer</a> [% dancer_version %]
```
这里修改了<%%>为[%%],其他的tt模版也要改。
然后运行perl cachemoni/bin/app.pl,启动3000端口的监听,访问一下如下:
<img src="/images/uploads/squid-monitor-demo.jpg" alt="" title="123" width="571" height="401" class="alignnone size-full wp-image-2505" />
这就算完成一个小的网页了,然后开始配置进apache:
```bashcpanm Plack::Handler::Apache2
wget http://search.cpan.org/CPAN/authors/id/P/PH/PHRED/mod_perl-2.0.5.tar.gz
tar zxvf mod_perl-2.0.5.tar.gz
cd mod_perl-2.0.5
perl Makefile.PL
然后会提示输入apxs的全路径:/usr/local/apache2/bin/apxs
make && make install
sed -i s'/LoadModule/i\LoadModule perl_module modules/mod_perl.so\' /usr/local/apache2/conf/httpd.conf
cat >> /usr/local/apache2/conf/httpd.conf <<EOF
<VirtualHost *:80>
ServerName dancer.test.china.com
DocumentRoot /www/html/cachemoni
<Directory /www/html/cachemoni>
AllowOverride None
Order allow,deny
Allow from all
<location />
SetHandler perl-script
PerlHandler Plack::Handler::Apache2
PerlSetVar psgi_app /www/html/cachemoni/bin/app.pl
EOF
/usr/local/apache2/bin/apachectl restart
```
访问一下,OK~
各种和webserver搭配的方法(其实就是两种:mod_perl和各种cgi)详见:<a href="http://search.cpan.org/~sukria/Dancer-1.3060/lib/Dancer/Deployment.pod">CPAN文档</a>
</div></body></dd></dt></dt></dl></center>
patch制作和使用
2011-06-23T00:00:00+08:00
linux
http://chenlinux.com/2011/06/23/make-patch-of-squid-snmp
<p>做运维几年没用过patch,说来也怪了~~趁着上一篇自己的小改动,熟悉一下这个命令的简单用法。<br />
首先是patch的制作,用diff命令。<br />
<code class="highlighter-rouge">bashtar zxvf squid-2.7.STABLE9.tar.gz
cp squid-2.7.STABLE9 squid-2.7.STABLE9-old && mv squid-2.7.STABLE9 squid-2.7.STABLE9-new
#然后按照上篇的内容修改squid-2.7.STABLE9-new/里的文件
diff -uNr squid-2.7.STABLE9-old squid-2.7.STABLE9-new > squid-snmp.patch</code><br />
这就完成了,好简单啊~来看看patch文件的内容吧:<br />
<code class="highlighter-rouge">cdiff -uNr squid-2.7.STABLE9-old/include/cache_snmp.h squid-2.7.STABLE9-new/include/cache_snmp.h
--- squid-2.7.STABLE9-old/include/cache_snmp.h 2006-09-22 10:49:24.000000000 +0800
+++ squid-2.7.STABLE9-new/include/cache_snmp.h 2011-06-23 13:25:04.000000000 +0800
@@ -125,6 +125,7 @@
MESH_PTBL_KEEPAL_S,
MESH_PTBL_KEEPAL_R,
MESH_PTBL_INDEX,
+ MESH_PTBL_CONN_OPEN,
MESH_PTBL_HOST,
MESH_PTBL_END
};
diff -uNr squid-2.7.STABLE9-old/src/snmp_agent.c squid-2.7.STABLE9-new/src/snmp_agent.c
--- squid-2.7.STABLE9-old/src/snmp_agent.c 2009-06-26 06:58:10.000000000 +0800
+++ squid-2.7.STABLE9-new/src/snmp_agent.c 2011-06-23 13:27:31.000000000 +0800
@@ -264,6 +264,11 @@
index,
ASN_INTEGER);
break;
+ case MESH_PTBL_CONN_OPEN:
+ Answer = snmp_var_new_integer(Var->name, Var->name_length,
+ p->stats.conn_open,
+ ASN_INTEGER);
+ break;
default:
*ErrP = SNMP_ERR_NOSUCHNAME;
break;
diff -uNr squid-2.7.STABLE9-old/src/snmp_core.c squid-2.7.STABLE9-new/src/snmp_core.c
--- squid-2.7.STABLE9-old/src/snmp_core.c 2008-05-05 07:23:13.000000000 +0800
+++ squid-2.7.STABLE9-new/src/snmp_core.c 2011-06-23 20:36:54.000000000 +0800
@@ -321,7 +321,7 @@
snmpAddNode(snmpCreateOid(LEN_SQ_MESH + 3, SQ_MESH, 1, 1, 15),
LEN_SQ_MESH + 3, snmp_meshPtblFn, peer_Inst, 0)),
snmpAddNode(snmpCreateOid(LEN_SQ_MESH + 2, SQ_MESH, 1, 2),
- LEN_SQ_MESH + 2, NULL, NULL, 15,
+ LEN_SQ_MESH + 2, NULL, NULL, 16,
snmpAddNode(snmpCreateOid(LEN_SQ_MESH + 3, SQ_MESH, 1, 2, 1),
LEN_SQ_MESH + 3, snmp_meshPtblFn, peer_InstIndex, 0),
snmpAddNode(snmpCreateOid(LEN_SQ_MESH + 3, SQ_MESH, 1, 2, 2),
@@ -351,6 +351,8 @@
snmpAddNode(snmpCreateOid(LEN_SQ_MESH + 3, SQ_MESH, 1, 2, 14),
LEN_SQ_MESH + 3, snmp_meshPtblFn, peer_InstIndex, 0),
snmpAddNode(snmpCreateOid(LEN_SQ_MESH + 3, SQ_MESH, 1, 2, 15),
+ LEN_SQ_MESH + 3, snmp_meshPtblFn, peer_InstIndex, 0),
+ snmpAddNode(snmpCreateOid(LEN_SQ_MESH + 3, SQ_MESH, 1, 2, 16),
LEN_SQ_MESH + 3, snmp_meshPtblFn, peer_InstIndex, 0))),
snmpAddNode(snmpCreateOid(LEN_SQ_MESH + 1, SQ_MESH, 2),
LEN_SQ_MESH + 1, NULL, NULL, 1,</code><br />
制作练完了,再练一次使用:<br />
<code class="highlighter-rouge">bashcd squid-2.7.STABLE9-old
mv ../squid-snmp.patch .
patch -p1 < squid-snmp.patch</code><br />
-p指定从那层目录开始,因为之前diff的时候顶层目录分别叫old和new,如果在其他地方时候的话,别人的目录肯定不会这么命名的,所以就往里进一层,然后用-p1来patch。<br />
然后more一下那三个文件,确认都修改了~<br />
最后回退patch:<br />
<code class="highlighter-rouge">bashpatch -R -p1 < squid-snmp.patch</code><br />
完成~</p>
给squid的snmp增加open_conn输出
2011-06-22T00:00:00+08:00
monitor
squid
C
snmp
http://chenlinux.com/2011/06/22/extend-open_conn-output-support-to-squid-snmp
<p>做反向代理的squid集群监控,在单机维护时,squidclient mgr:server_list里的OPEN CONNS是经常看的一项数据,不过在开启snmp支持后,在mib里却没有找到相关的数据。还一度怀疑是不是cachePeerKeepAlRecv或者cachePeerKeepSent。今天想起来去src里grep了一把源码,顺利的在squid/src/neighbors.c里看到了OPEN CONNS等数据的来源,如下:<br />
<code class="highlighter-rouge">cstatic void
dump_peers(StoreEntry * sentry, peer * peers)
{
peer *e = NULL;
……
for (e = peers; e; e = e->next) {
……
storeAppendPrintf(sentry, "OPEN CONNS : %d\n", e->stats.conn_open);
……
storeAppendPrintf(sentry, "keep-alive ratio: %d%%\n",
percent(e->stats.n_keepalives_recv, e->stats.n_keepalives_sent));</code><br />
然后在squid/src/snmp_agent.c里看到了这些数据的snmp输出,如下:<br />
<code class="highlighter-rouge">cvariable_list *
snmp_meshPtblFn(variable_list * Var, snint * ErrP)
{
variable_list *Answer = NULL;
struct in_addr *laddr;
int loop, index = 0;
char *cp = NULL;
peer *p = NULL;
int cnt = 0;
……
switch (Var->name[LEN_SQ_MESH + 2]) {
case MESH_PTBL_NAME:
cp = p->name;
Answer = snmp_var_new(Var->name, Var->name_length);
……
case MESH_PTBL_KEEPAL_R:
Answer = snmp_var_new_integer(Var->name, Var->name_length,
p->stats.n_keepalives_recv,
SMI_COUNTER32);
break;
case MESH_PTBL_INDEX:
Answer = snmp_var_new_integer(Var->name, Var->name_length,
index,
ASN_INTEGER);
break;
default:
*ErrP = SNMP_ERR_NOSUCHNAME;
break;
}
return Answer;
}</code><br />
一对比,发现确实没有stats.conn_open输出……<br />
好在这个比较简单,稍微改一下,就能搞出来:<br />
1、修改squid/include/cache_snmp.h如下:<br />
<code class="highlighter-rouge">cenum { /* cachePeerTable */
……
MESH_PTBL_CONN_OPEN, /*新增这个*/
MESH_PTBL_HOST,
MESH_PTBL_END
};</code><br />
2、修改squid/src/snmp_core.c如下:<br />
<code class="highlighter-rouge">c
void
snmpInit(void)
{
……
snmpAddNode(snmpCreateOid(LEN_SQ_MESH + 2, SQ_MESH, 1, 2),
/* LEN_SQ_MESH + 2, NULL, NULL, 15,这里改成16,大概在324行,通过原来的MIB知道有15的地方就两个,peer的是后一个 */
LEN_SQ_MESH + 2, NULL, NULL, 16,
……
snmpAddNode(snmpCreateOid(LEN_SQ_MESH + 3, SQ_MESH, 1, 2, 15),
LEN_SQ_MESH + 3, snmp_meshPtblFn, peer_InstIndex, 0),
snmpAddNode(snmpCreateOid(LEN_SQ_MESH + 3, SQ_MESH, 1, 2, 16), /*新增这个16*/
LEN_SQ_MESH + 3, snmp_meshPtblFn, peer_InstIndex, 0))),</code><br />
3、修改squid/src/snmp_agent.c如下:<br />
<code class="highlighter-rouge">c……
case MESH_PTBL_INDEX:
Answer = snmp_var_new_integer(Var->name, Var->name_length,
index,
ASN_INTEGER);
break;
/*新增下面这段,case的内容在第1步cache_snmp.h里增加了;stats.conn_open由之前grep的结果得知;INTEGER是数值类型,照抄RTT的即可*/
case MESH_PTBL_CONN_OPEN:
Answer = snmp_var_new_integer(Var->name, Var->name_length,
p->stats.conn_open,
ASN_INTEGER);
break;
</code><br />
4、重新编译squid,然后用snmpwalk获取数据观察:<br />
<code class="highlighter-rouge">bash[root@naigos myops]# snmpwalk -v 2c -c cacti_china 10.168.168.69 .1.3.6.1.4.1.3495.1.5.1.2 -Cc | tail
SNMPv2-SMI::enterprises.3495.1.5.1.2.13.3 = Counter32: 0
SNMPv2-SMI::enterprises.3495.1.5.1.2.14.1 = INTEGER: 1
SNMPv2-SMI::enterprises.3495.1.5.1.2.14.2 = INTEGER: 2
SNMPv2-SMI::enterprises.3495.1.5.1.2.14.3 = INTEGER: 3
SNMPv2-SMI::enterprises.3495.1.5.1.2.15.1 = INTEGER: 3
SNMPv2-SMI::enterprises.3495.1.5.1.2.15.2 = INTEGER: 5
SNMPv2-SMI::enterprises.3495.1.5.1.2.15.3 = INTEGER: 6
SNMPv2-SMI::enterprises.3495.1.5.1.2.16.1 = STRING: "10.168.170.43"
SNMPv2-SMI::enterprises.3495.1.5.1.2.16.2 = STRING: "10.168.168.73"
SNMPv2-SMI::enterprises.3495.1.5.1.2.16.3 = STRING: "10.168.168.122"</code><br />
原来的SNMPv2-SMI::enterprises.3495.1.5.1.2.15.1 = STRING: “10.168.170.43”变成了SNMPv2-SMI::enterprises.3495.1.5.1.2.16.1 = STRING: “10.168.170.43”,而SNMPv2-SMI::enterprises.3495.1.5.1.2.15.1 = INTEGER: 3就是需要的open_conn数据了!</p>
mysql测试小工具mybench试用
2011-06-14T00:00:00+08:00
testing
perl
MySQL
http://chenlinux.com/2011/06/14/intro-mybench
<p>小型的mysql测试工具,主要有自带的mysqlslap、super-smack和mybench。嗯,我这里的小型的意思是指工具安装过程简单。<br />
mysqlslap的使用方法遍地都是,就不先详细写了。根据个人偏好写写mybench吧,毕竟是perl的。<br />
安装很简单,如下:<br />
<code class="highlighter-rouge">bashcpanm DBI DBD::mysql Time::HiRes
wget http://jeremy.zawodny.com/mysql/mybench/mybench-1.0.tar.gz
tar zxvf mybench-1.0.tar.gz
cd mybench-1.0
perl MakeFile.PL && make && make install</code><br />
但是使用就不是太简单了——mysqlslap会自己生成(-a选项)sql,super-smack则带了一个gen-data程序生成数据然后自动导入,但是mybench没有,所以只能自己搞定数据。<br />
不过mybench还是自己生成了一个测试模版的脚本在/usr/bin/bench_example,很简单的就知道怎么做了。<br />
example如下:<br />
```perl#!/usr/bin/perl -w</p>
<p>eval ‘exec /usr/bin/perl -w -S $0 ${1+”$@”}’<br />
if 0; # not running under some shell</p>
<p>use strict;<br />
use MyBench;<br />
use Getopt::Std;<br />
use Time::HiRes qw(gettimeofday tv_interval);<br />
use DBI;</p>
<p>my %opt;<br />
Getopt::Std::getopt(‘n:r:h:’, \%opt);<br />
#这是我见过的最hardcode的perl脚本了(呃,除了我自己写的垃圾),连db库、用户名、密码都不给运行参数的<br />
my $num_kids = $opt{n} || 10;<br />
my $num_runs = $opt{r} || 100;<br />
my $db = “test”;<br />
my $user = “test”;<br />
my $pass = “”;<br />
my $port = 3306;<br />
my $host = $opt{h} || “192.168.0.1”;<br />
my $dsn = “DBI:mysql:$db:$host;port=$port”;</p>
<p>my $callback = sub<br />
{<br />
my $id = shift;<br />
my $dbh = DBI->connect($dsn, $user, $pass, { RaiseError => 1 });<br />
#为测试准备的请求,测select就写select,测insert就写insert呗~<br />
#如果不修改,也就是说测试用的是test.mytable表,而且必须有一个列叫id<br />
my $sth = $dbh->prepare(“SELECT * FROM mytable WHERE ID = ?”);</p>
<div class="highlighter-rouge"><pre class="highlight"><code>my $cnt = 0;
my @times = ();
## wait for the parent to HUP me
local $SIG{HUP} = sub { };
sleep 600; #脚本定义的每个进程执行多少次请求
while ($cnt < $num_runs)
{
my $v = int(rand(100_000));
## time the query
my $t0 = [gettimeofday]; #真正的执行sql请求,通过上面的rand知道,之前准备的test.mytable的id列必须是int格式,同时不少于10w行(又一处hard)
$sth->execute($v); #通过前后两次gettimeofday获得sql的exec耗时
my $t1 = tv_interval($t0, [gettimeofday]); #完成一次请求执行,加入数组
push @times, $t1;
$sth->finish();
$cnt++;
}
## cleanup
$dbh->disconnect(); #计算本进程全部请求的各项数据,几个大小和均来自MyBench模块
my @r = ($id, scalar(@times), min(@times), max(@times), avg(@times), tot(@times));
return @r; }; #将上面这个函数交给MyBench模块的fork_and_work执行,即并发指定数量请求,返回总的结果 my @results = MyBench::fork_and_work($num_kids, $callback); #计算总的数据 MyBench::compute_results('test', @results);
</code></pre>
</div>
<p>exit;</p>
<p><strong>END</strong><code class="highlighter-rouge">
然后看看/usr/lib/perl5/site_perl/5.8.8/MyBench.pm,主要内容就是fork和compute:
</code>perlpackage MyBench;<br />
use strict;</p>
<p>$main::VERSION = ‘1.0’;</p>
<p>use Exporter;<br />
@MyBench::ISA = ‘Exporter’;<br />
#导出求最大值、最小值、平均值、综合值的函数给外面用<br />
@MyBench::EXPORT = qw(max min avg tot);</p>
<p>sub fork_and_work($$)<br />
{<br />
#关闭输出缓冲<br />
$|=1;</p>
<div class="highlighter-rouge"><pre class="highlight"><code>use strict;
use IO::Pipe;
use IO::Select;
$SIG{CHLD} = 'IGNORE'; ## let the kids die
my $kids_to_fork = shift;
my $callback = shift;
my $num_kids = 0;
my @pipes = ();
my @pids = ();
my $pid = undef;
print "forking: ";
while ($num_kids < $kids_to_fork)
{ #用IO::Pipe管道方式来传递父子进程的信息
my $pipe = new IO::Pipe; #fork进程开始
if ($pid = fork())
{
## parent
$num_kids++; #每fork完成一个打印一个+号
print "+"; #从管道中读取数据
$pipe->reader();
push @pipes, $pipe;
push @pids, $pid;
}
elsif (defined $pid)
{
## child #打开管道写入数据的功能
$pipe->writer(); #执行select_example脚本传入的mysql请求测试函数
my @result = $callback->($num_kids); #把结果写入管道
print $pipe "@result\n"; #关闭管道
$pipe->close();
exit 0;
}
else
{
print "fork failed: $!\n";
}
}
print "\n";
## give them a bit of time to setup
my $time = int($num_kids / 10) + 1;
print "sleeping for $time seconds while kids get ready\n";
sleep $time;
</code></pre>
</div>
<p>#发送SIGHUP信号给callback函数<br />
kill 1, @pids;</p>
<div class="highlighter-rouge"><pre class="highlight"><code>## collect the results
my @results;
print "waiting: "; #从管道中读取数据到数组
for my $pipe (@pipes)
{
my $data = <$pipe>;
push @results, $data;
$pipe->close();
print "-";
}
print "\n";
return @results; }
</code></pre>
</div>
<p>sub compute_results(@)<br />
{<br />
my $name = shift;<br />
my $recs = 0;<br />
my ($Cnt, $Min, $Max, $Avg, $Tot, @Min, @Max);</p>
<div class="highlighter-rouge"><pre class="highlight"><code>while (@_)
{
## 6 elements per record
my $rec = shift; chomp $rec;
my ($id, $cnt, $min, $max, $avg, $tot) = split /\s+/, $rec;
$Cnt += $cnt;
$Avg += $avg;
$Tot += $tot;
push @Min, $min;
push @Max, $max;
$recs++;
}
$Avg = $Avg / $recs;
$Min = min(@Min);
$Max = max(@Max);
my $Qps = $Cnt / ($Tot / $recs);
print "$name: $Cnt $Min $Max $Avg $Tot $Qps\n";
print " clients : $recs\n";
print " queries : $Cnt\n";
print " fastest : $Min\n";
print " slowest : $Max\n";
print " average : $Avg\n";
print " serial : $Tot\n";
print " q/sec : $Qps\n"; }
</code></pre>
</div>
<h2 id="some-numerical-helper-functions-for-arrays">some numerical helper functions for arrays</h2>
<p>sub max<br />
{<br />
my $val = $<em>[0];<br />
for (@</em>)<br />
{<br />
if ($_ > $val) { $val = $_; }<br />
}<br />
return $val;<br />
}</p>
<p>sub min<br />
{<br />
my $val = $<em>[0];<br />
for (@</em>)<br />
{<br />
if ($_ < $val) { $val = $_; }<br />
}<br />
return $val;<br />
}</p>
<p>sub avg<br />
{<br />
my $tot;<br />
for (@<em>) { $tot += $</em>; }<br />
return $tot / @_;<br />
}</p>
<p>sub tot<br />
{<br />
my $tot;<br />
for (@<em>) { $tot += $</em>; }<br />
return $tot;<br />
}</p>
<p>1;<code class="highlighter-rouge">
好了,开始准备数据,比较懒,直接用super-smack的gen-data先出了一些./gen-data -n 100000 -f %n,%80-12s%12n,%512-512s,%d > /root/data,然后进mysql里执行:
</code>mysql<br />
USE test;<br />
CREATE TABLE mytable (id INT(11) NOT NULL AUTO_INCREMENT, col1 CHAR(100), col2 CHAR(100), col3 INT(11), PRIMARY KEY (id) )ENGINE=InnoDB DEFAULT CHARSET=utf8;<br />
LOAD DATA LOCAL INFILE ‘data’ REPLACE INTO TABLE ‘mytable’ FIELDS TERMINATED BY ‘,’ LINES TERMINATED BY ‘\n’;<br />
INSERT INTO mytable (col1,col2,col3) SELECT col1,col2,col3 FROM mytable;```<br />
最后执行./select_bench -h 10.168.170.92 -n 10 -r 1000就能看到结果了:<br />
forking: ++++++++++<br />
sleeping for 2 seconds while kids get ready<br />
waiting: ———-<br />
test: 10000 0.00017 0.006809 0.0010413514 10.413514 9602.9063772325<br />
clients : 10<br />
queries : 10000<br />
fastest : 0.00017<br />
slowest : 0.006809<br />
average : 0.0010413514<br />
serial : 10.413514<br />
q/sec : 9602.9063772325</p>
java的中文支持
2011-06-10T00:00:00+08:00
linux
http://chenlinux.com/2011/06/10/java-chinese-support
<p>往论坛上传图片,有的图片上有中文字,却显示成方框。求助了一下度娘,快速解决。记录一下:<br />
* 第一步,在windows下找到simsun.ttc文件,嗯,ntfs系统下强力推荐everything小工具一个;<br />
* 第二步,上传simsun.ttc到服务器的/usr/share/fonts/zh_CN/下,其他路径也行,不过这个路径比较通用;<br />
* 第三步,在$JAVA_HOME/jrp/lib/下创建fontconfig.properties.zh文件,原型格式可见同目录下的fontconfig.properties.src。<br />
fontconfig.properties.zh文件相关内容如下:<br />
<code class="highlighter-rouge">java
allfonts.chinese-gbk=-misc-simsun-medium-r-normal--*-%d-*-*-p-*-gbk-0
allfonts.chinese-gb2312=-misc-simsun-medium-r-normal--*-%d-*-*-p-*-gb2312.1980-0
sequence.allfonts.GB18030=latin-1,chinese-gbk,chinese-cn-iso10646
sequence.allfonts.GBK=latin-1,chinese-gbk
sequence.allfonts.GB2312=latin-1,chinese-gb2312
sequence.allfonts.UTF-8.ko.KR=latin-1,korean,japanese-x0208,japanese-x0201,chinese-gbk
sequence.allfonts.UTF-8.ja.JP=latin-1,japanese-x0208,japanese-x0201,chinese-gbk,korean
sequence.fallback=lucida,chinese-big5,chinese-gbk,japanese-x0208,korean
filename.-misc-simsun-medium-r-normal--*-%d-*-*-p-*-gbk-0=/usr/share/fonts/zh_CN/simsun.ttc
filename.-misc-simsun-medium-r-normal--*-%d-*-*-p-*-gb2312.1980-0=/usr/share/fonts/zh_CN/simsun.ttc
awtfontpath.chinese-gb2312=/usr/share/fonts/zh_CN
awtfontpath.chinese-gbk=/usr/share/fonts/zh_CN
</code><br />
很简单的一件小事儿,一来作个记录,二来测试微博同步——从百度统计看我可怜的一点点访问都来自微博……</p>
perl模块Statistics::Descriptive
2011-06-09T00:00:00+08:00
perl
http://chenlinux.com/2011/06/09/statistics-descriptive-module-of-perl
<p>今天写基调测试报告,需要从原始的ping延时和丢包率数据中自己计算标准方差以评估波动性(直接运行ping命令可见,不过基调报告里没有)。<br />
方差是各个数据与其平均数的差的平方的平均数。标准差(均方差)则是方差的算术平方根。<br />
这个时候可以打开excel……不过作为excel只会填文字的人,只好打开CPAN来解决问题了~<br />
<code class="highlighter-rouge">perl#!/usr/bin/perl -w
use Statistics::Descriptive;
use strict;
open FH,'<','data';
my $data={};
while(<FH>){
my @F = split;
push @{$data->{"快网延时"}}, $F[3];
push @{$data->{"快网丢包"}}, $F[4];
push @{$data->{"森华延时"}}, $F[6];
push @{$data->{"森华丢包"}}, $F[7];
push @{$data->{"帝联延时"}}, $F[9];
push @{$data->{"帝联丢包"}}, $F[10];
}
close FH;
my $stat = Statistics::Descriptive::Full->new();
foreach my $key (sort keys %{$data}) {
$stat->add_data(@{$data->{"$key"}});
print $key."\t".$stat->standard_deviation(),"\n";
$stat->clear();
}</code><br />
记住一定要clear,不然的话add_data会接着上一次的加,然后数据就错了。</p>
spread试验
2011-06-09T00:00:00+08:00
perl
http://chenlinux.com/2011/06/09/intro-spread
<p>spread还是半年前的时候偶然看到的,一直没有试过。前段时间用gearman收集集群日志时,发现gearman的方式,worker不会知道client来自哪里,一条job只会一个worker来做,比较适合做分布式计算,但相比我最初设想的实时系统管理需求,还是有一定距离。于是重新翻出来spread,感觉可以根据应用系统设置不同的group,然后统一再由一个回收结果的group即可。于是有了如下试验:</p>
<ul>
<li>spread安装配置:</li>
</ul>
<p><code class="highlighter-rouge">bashwget http://www.spread.org/download/spread-src-4.1.0.tar.gz
tar zxvf spread-src-4.1.0.tar.gz
cd spread-src-4.1.0
./configure --prefix=/usr/local/spread && make && make install
cat > /usr/local/spread/etc/spread.conf << EOF
Spread_Segment 10.1.171.255:4804 {
ct-142 10.1.168.142
ct-94 10.1.168.94
ct-241 10.1.168.241
ct-156 10.1.168.156
ct-70 10.1.170.70
cnc-64 10.1.169.64
cnc-80 10.1.169.80
cnc-72 10.1.169.72
cnc-58 10.1.169.58
}
EOF
groupadd spread
useradd -g spread spread
mkdir -p /var/run/spread
chown spread:spread /var/run/spread
echo '/usr/local/spread/lib' > /etc/ld.conf.d/spread.conf && ldconfig
#必须用-n指定配置文件中定义好了的servername;
#奇怪的是网上别的文章都指出这些配置要同时写入hosts,但我没写也一样用了
/usr/local/spread/sbin/spread -c /usr/local/spread/etc/spread.conf -n ct-156 &</code></p>
<ul>
<li>perl的spread使用</li>
</ul>
<p>CPAN上有很多关于spread的模块,试了几个后,选中了Spread::Messaging::Content。使用如下:<br />
```perl#!/usr/bin/perl -w<br />
use Spread::Messaging::Content;<br />
use Event;</p>
<p>$spread = Spread::Messaging::Content->new(<br />
-port => “4804”,<br />
-timeout => “10”,<br />
-host => “10.1.168.156”,<br />
);<br />
$spread->join_group(“test”);<br />
#当spread的group存在了filedescriptor后,执行子函数;<br />
#$spread->fd来自Spread::Messaging::Transport,这个module是Spread::Messaging::Content自动加载调用的<br />
Event->io(fd => $spread->fd, cb => \&put_output);<br />
Event::loop();</p>
<p>sub put_output {<br />
$spread->recv();<br />
printf(“Sender : %s\n”, $spread->sender);<br />
printf(“Groups : %s\n”, join(‘,’, @{$spread->group}));<br />
printf(“Message : %s\n”, ref($spread->message) eq “ARRAY” ? <br />
join(‘,’, @{$spread->message}) :<br />
$spread->message);<br />
}<code class="highlighter-rouge">
</code>perl#!/usr/bin/perl -w<br />
use Spread::Messaging::Content;<br />
$spread = Spread::Messaging::Content->new(<br />
-port => “4804”,<br />
-timeout => “10”,<br />
-host => “10.1.168.156”,<br />
);</p>
<p>$spread->group(“test2”);<br />
$spread->type(“0”);<br />
$spread->message(“cooking with fire”);<br />
$spread->send();```</p>
<ul>
<li>spread自带的spuser使用</li>
</ul>
<p><code class="highlighter-rouge">bash/usr/local/spread/bin/spuser -s 4804
j test
m test</code></p>
tmpfs的inode问题
2011-06-09T00:00:00+08:00
linux
http://chenlinux.com/2011/06/09/inode-problem-of-tmpfs
<p>一些squid服务器为了强调加速效果,使用tmpfs来做cache_dir。刚开始运行的时候也嗖嗖的,不过没过一两天,mgr:info就看到缓存命中率急剧下降,字节命中率甚至只剩下10%左右!检查了多次配置,绝对没有问题,但同样的url,曾经一分钟几百次的HIT,现在一分钟几百次MISS……<br />
df看,不管是tmpfs,还是logs所在的目录,都才用了不到30%。最后想起来df -i看了下,果然,tmpfs的inode使用率100%了!<br />
赶紧remount了一次,解决了问题。但不是根本出路。还是得想办法搞定这个inode。<br />
在linux代码说明里找到了关于tmpfs的文档(/usr/src/linux/Documentation/filesystems/tmpfs.txt):<br />
tmpfs has three mount options for sizing:<br />
……<br />
nr_inodes: The maximum number of inodes for this instance. <strong>The default
is half of the number of your physical RAM pages</strong>, or (on a<br />
machine with highmem) the number of lowmem RAM pages,<br />
whichever is the lower.<br />
These parameters accept a suffix k, m or g for kilo, mega and giga and<br />
can be changed on remount. The size parameter also accepts a suffix %<br />
to limit this tmpfs instance to that percentage of your physical RAM:<br />
<strong>the default, when neither size nor nr_blocks is specified, is size=50%</strong></p>
<div class="highlighter-rouge"><pre class="highlight"><code>If nr_blocks=0 (or size=0), blocks will not be limited in that instance;
<strong>if nr_inodes=0, inodes will not be limited.</strong> It is generally unwise to
mount with such options, since it allows any user with write access to
use up all the memory on the machine; but enhances the scalability of
that instance in a system with many cpus making intensive use of it. linux默认的RAM page大小是4k,好了来计算一下吧。 ```bash[root@bbs_squid4 ~]# df -i|awk '/tmpfs/{print $2}' 504912 [root@bbs_squid4 ~]# free -k|awk '/Mem/{print $2/4/2}' 504912``` 果然如此! 那么真正的解决办法也就有了: ```bash[root@localhost ~]# mount -t tmpfs -o size=2000M,mode=777,nr_inodes=0 tmpfs /tmpfs [root@localhost ~]# df -i|grep tmpfs tmpfs 0 0 0 - /tmpfs```
</code></pre>
</div>
用gearman汇总多台服务器的回滚日志
2011-06-03T00:00:00+08:00
monitor
gearman
perl
http://chenlinux.com/2011/06/03/aggregate-multi-servers-rollback-log-by-gearmand
<p>gearman其实不是重点,因为我就是抄了一遍perldoc的样例而已。关键在服务器上的log4j日志是回滚的,所以需要配合回滚(猜测log4j的DailyRollingFile回滚方式类似mv resin.log resin.log-ymd && reload,这样在回滚后,FH还在resin.log-ymd上,就读不到新日志了)重启FH。<br />
另:tail命令有个参数-F/–follow=name,可以锁定文件名而不是文件描述符,不知道这个功能是怎么做到的?<br />
一步一步来:</p>
<ul>
<li>jobserver</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code>cpan -i Gearman::Server
gearmand -d -L 10.168.170.25 -p 7003
</code></pre>
</div>
<ul>
<li>worker</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code>cpan -i Gearman::Worker
</code></pre>
</div>
<p>然后看日志的脚本,其实也就是样例:<br />
```perl<br />
#!/usr/bin/perl -w<br />
use Gearman::Worker;</p>
<p>my $worker = Gearman::Worker->new;<br />
$worker->job_servers(‘10.1.1.25:7003’);<br />
$worker->register_function( watchlog => \&watchlog );<br />
$worker->work while 1;</p>
<p>sub watchlog {<br />
my $job = shift;<br />
print $job->arg,”\n”;<br />
}<br />
```</p>
<ul>
<li>client(也就是多台resin服务器上)</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code>cpan -i Gearman::Client
</code></pre>
</div>
<p>然后是脚本:<br />
```perl<br />
#!/usr/bin/perl -w<br />
use Gearman::Client;<br />
use POSIX qw(strftime);</p>
<p>my $client = Gearman::Client->new;<br />
$client->job_servers(‘10.1.1.25:7003’);</p>
<p>&read;<br />
while (1) {<br />
open FH1,’<’,’/tmp/pid.txt’;<br />
my $childpid = <fh1>;
close FH1;
my $date=strftime("%H:%M:%S",localtime);
if ($date eq '00:00:00') {
kill 9,$childpid;
sleep 1;
&read;
}
}</fh1></p>
<p>sub read {<br />
my $pid = fork();<br />
if ($pid == 0) {<br />
#这个<script type="math/tex">不能直接赋值出去,所以采用文件方式,以后研究一下pipe啊之类的办法。
open FH,'>','/tmp/pid.txt';
print FH</script>;<br />
close FH;<br />
open (FD,’<’,’resin.log’) or die $!;<br />
while (1) {<br />
my $log = <fd>;
sleep 1 and next unless $log;
chomp $log;
$client->dispatch_background('watchlog',$log);
}
close FD;
}
}
```</fd></p>
HTML::Template试用
2011-06-01T00:00:00+08:00
perl
http://chenlinux.com/2011/06/01/intro-html-template
<p>给我自己的学习计划做个开头,从html::template开始试用。<br />
首先利用上上篇的nmap.pl脚本,提取一些数据,然后展示在页面上。<br />
cgi脚本如下:<br />
<code class="highlighter-rouge">perl#!/usr/bin/perl -w
use HTML::Template;
use XML::Simple;
use Net::MySQL;
#定期执行这个
#system("nmap -n -p 22,5666 10.168.168.0/23 10.168.170.0/24 -oX output.xml");
my $text = XMLin("output.xml");
#读取html模版
my $temp = HTML::Template->new(filename => '../template/html/server.tmpl');
my $localhost = '127.0.0.1';
my @array = ();
my $i = 0;
my $hash = {};
while ( $text->{host}->[$i] ) {
#因为新增了ssh端口扫描,所以xml解析和前例稍有不同
my $ssh_state = $text->{host}->[$i]->{ports}->{port}->[0]->{state}->{state};
my $nrpe_state = $text->{host}->[$i]->{ports}->{port}->[1]->{state}->{state};
my $ip = ref($text->{host}->[$i]->{address}) eq 'ARRAY' ? $text->{host}->[$i]->{address}->[0]->{addr} : $text->{host}->[$i]->{address}->{addr};
my $mac = ref($text->{host}->[$i]->{address}) eq 'ARRAY' ? $text->{host}->[$i]->{address}->[1]->{addr} : '00:1E:C9:E6:E1:7C';
$i++;
my $channel = &amp;mysql_query($mac);
#将ip按照频道排成列表,每个ip存有ssh和nrpe状态
if ( exists $hash->{$channel} ) {
push @{$hash->{$channel}}, { 'IP' => $ip, 'SSH' => $ssh_state, 'NRPE' => $nrpe_state, };
} else {
$hash->{$channel}->[0] = { 'IP' => $ip, 'SSH' => $ssh_state, 'NRPE' => $nrpe_state, };
}
}
#将上面while生成的hash转成HTML::Template认可的array,不过array的单个元素可以是hash
foreach my $key( keys %{$hash} ) {
my $onechannel = {};
$onechannel->{"CHANNEL"} = $key;
my $j = 0;
foreach my $ip( @{$hash->{$key}} ) {
$onechannel->{"IP_LOOP"}->[$j] = $ip;
$j++;
}
push @array, $onechannel;
}
#将array传递给之前定义的html模版
#注意:不管是param还是@array里,所有的key必须都在tmpl里使用,冗余也会报错
$temp->param(CHANNEL_LOOP => \@array);
#输出成html格式
print "Content-Type: text/html\n\n", $temp->output;
#这段没什么说的,根据mac获取频道
sub mysql_query {
my $mac = shift;
my $mysql = Net::MySQL->new( hostname => $localhost,
database => 'myops',
user => 'myops',
password => 'myops',
);
$mysql->query("select channel from myhost where mac='$mac'");
&amp;alert("New server") unless $mysql->has_selected_record;
my $a_record_iterator = $mysql->create_record_iterator();
while (my $record = $a_record_iterator->each) {
return $record->[0];
};
}
#留着后续继续处理
sub alert {
print @_,"\n";
}</code><br />
然后是template文件server.tmpl:<br />
```html</p>
<html>
<head>
<title>Server Plate</title>
</head>
<body>
<table width="100%" cellspacing="0" cellpadding="0" border="1">
<!--TMPL_LOOP循环格式,使用的是array里channel_loop的每个元素-->
<tmpl_loop name="CHANNEL_LOOP">
<tr>
<!--根据本层loop中的某个元素的channel的value开始表格的一行-->
<th><center><tmpl_var name="CHANNEL">
<!--本层loop中另一个元素ip_loop,也是array格式,所以继续循环,每个元素使用一列-->
<tmpl_loop name="IP_LOOP">
<td valign=top><center>
//根据本层loop的ssh情况选择显示哪个图标;TMPL_IF只能判断key的真假,所以用js
<script type="text/javascript">
if ('<TMPL_VAR NAME="SSH">' == 'open') {
document.write("<img src='../template/images/unlock_server.png'>");
} else {
document.write("<img src='../template/images/desable_server.png'>");
}
</script>
<!--显示第二层loop里元素的几个value-->
<hr />nrpe:<tmpl_var name="NRPE"><hr />ssh :<tmpl_var name="SSH"><hr /><tmpl_var name="IP">
<!--结束里层loop,即完成一行表格-->
<!--结束顶层loop,即完成表格-->
<br /><br /><br /><center>
```
用apache分别发布cgi目录和静态目录。然后访问一下;OK。
</center></tmpl_var></tmpl_var></tmpl_var></center></tmpl_loop></tmpl_var></center></th></tr></tmpl_loop></table></body></html>
nginx两个小测试(perl_set/image_filter)
2011-05-30T00:00:00+08:00
nginx
perl
http://chenlinux.com/2011/05/30/intro-perl_set-image_filter
<p>第一个测试,关于http_perl_module。之前写过一篇关于nginx忽略大小写的博文,今天被朋友问上门来,url是类似/Upload/Dir/2011/123_D.jpg的形式。如果单纯的lc($r->uri),得到的url会变成/upload/dir/2011/123_d.jpg,目录是不存在的。所以要稍微改进一下。如下:<br />
<code class="highlighter-rouge">perl perl_set $url '
sub {
my $r = shift;
return $1.lc($2) if ($r->uri =~ m/^(.+\/)([^\/]+)$/);
return $r->uri;
}
';</code><br />
这样就行了。</p>
<p>另一个测试,关于http_image_filter_module。配置语句很简单,就一行image_filter [size|resize|corp] wight height;就行了——如果图片太大,那还要加大image_filter_buffer,默认1M,大于这个大小的图片就不会缩略了。<br />
比如配置如下:<br />
<code class="highlighter-rouge">nginx
location / {
root /var/www/html;
index index.html index.htm;
}
location ~* ^/small/w_(\d+)/h_(\d+)/(.*)$ {
rewrite /small/w_(\d+)/h_(\d+)/(.*)$ /$3 break;
image_filter resize $1 $2;
root /var/www/html;
index index.html index.htm;
}</code><br />
这样通过/small/w_100/h_50/path/to/text.jpg,就能访问到/path/to/text.jpg的100*50大小的缩略图了。<br />
如果只需要修改h或者w,其他的等比缩略,把另一项写成’-‘即可。<br />
其他参数介绍:test,返回是否真的是图片;corp,截取图片的一部分;size,以json格式返回图片的长宽数据。</p>
nmap扫描结果xml解析脚本
2011-05-27T00:00:00+08:00
monitor
nmap
perl
http://chenlinux.com/2011/05/27/analyze-the-xml-which-nmap-output
<p><code class="highlighter-rouge">perl#!/usr/bin/perl -w
use XML::Simple;
use Net::MySQL;
system("nmap -n -p 5666 10.1.1.0/23 10.1.3.0/24 -oX output.xml");
my $text = XMLin("output.xml");
my $i = 0;
while ( $text->{host}->[$i] ) {
my $nrpe = $text->{host}->[$i]->{ports}->{port}->{state}->{state};
#因为在扫描到本机的时候,是没有mac的,所以到本机时不是ARRAY而是HASH
my $ip = ref($text->{host}->[$i]->{address}) eq 'ARRAY' ? $text->{host}->[$i]->{address}->[0]->{addr} : $text->{host}->[$i]->{address}->{addr};
my $mac = ref($text->{host}->[$i]->{address}) eq 'ARRAY' ? $text->{host}->[$i]->{address}->[1]->{addr} : '00:1E:C9:E6:E1:7C';
&mysql_query($ip, $mac, $nrpe);
$i++;
}
sub mysql_query {
my ($ip, $mac, $nrpe) = @_;
my $mysql = Net::MySQL->new( hostname => '10.1.1.25',
database => 'myops',
user => 'myops',
password => 'myops',
);
$mysql->query(
"insert into myhost (intranet, mac, monitorstatus) values ('$ip', '$mac', '$nrpe')"
);
}</code><br />
小脚本一个,扫描内网网段内存活的机器,获取其MAC地址,以及nrpe端口情况。后期再配合myhost里的system,如果是linux(其实用nmap -O也可以获取system,但是结果不准,耗时还特别长,200台机器花10分钟),但monitorstatus还是closed的,就expect上去安装nrpe,嗯~~</p>
续上:合并纯真ip段
2011-05-20T00:00:00+08:00
monitor
perl
http://chenlinux.com/2011/05/20/merge-iplist-of-qqwry
<p>上篇提到纯真ip库有很多行是浪费的,比如下面这种:<code class="highlighter-rouge">yaml
223.214.0.0 223.215.255.255 安徽省 电信
223.216.0.0 223.219.255.255 日本
223.220.0.0 223.220.162.1 青海省 电信
223.220.162.2 223.220.162.2 青海省海东地区 平安县九歌网吧
223.220.162.3 223.221.255.255 青海省 电信</code><br />
很简单的223.220.0.0-223.221.255.255段,却被拆成了三行。于是在通过起始ip结束ip计算子网之前,还需要合并一下这些ip段。<br />
因为涉及ip比对,所以第一反应想到了mysql里有的inet_aton函数,去CPAN上搜了一下,发现有NetAddr::IP::Util模块有inet_aton函数,结果一用,发现居然生成的不是数字……于是从网上找到了pack的办法,如下:<br />
<code class="highlighter-rouge">perl#!/usr/bin/perl -w
while(<>){
next unless $_ =~ /^(\S+)\s+(\S+)\s+(\S+)/;
my $low = unpack('N',(pack('C4',(split( /\./,$1)))));
#下面这行是IP::QQWry模块里的写法
# print $1 * 256**3 + $2 * 256**2 + $3 * 256 + $4 if $1 =~ /(\d+)\.(\d+)\.(\d+)\.(\d+)/;;
my $high = unpack('N',(pack('C4',(split( /\./,$2)))));
next if $low == $high;
my $addr = $3;
unless ( $hash->{$addr}->{high}->[0] ) {
$hash->{$addr}->{low}->[0] = $low;
$hash->{$addr}->{high}->[0] = $high;
next;
};
#如果中间就隔几个ip的,可以无视之,合并就是了……
if ( $low - $hash->{$addr}->{high}->[0] < 16 ) {
$hash->{$addr}->{high}->[0] = $high;
next;
};
unshift @{$hash->{$addr}->{low}}, $low;
unshift @{$hash->{$addr}->{high}}, $high;
};
foreach $addr ( keys %{$hash} ) {
my $i = 0;
while ( $hash->{$addr}->{low}->[$i] ) {
print $addr . "\t" . &nota($hash->{$addr}->{low}->[$i]) . "\t" . &nota($hash->{$addr}->{high}->[$i]) , "\n";
$i++;
}
};
sub nota {
my $aton = shift;
@a = unpack('C4',(pack('N',$aton)));
return (join "\.",@a);
};</code><br />
pack真复杂,基本看不懂perldoc,唉……</p>
<p>最后汇报一下运行结果:<br />
合并前一共428452行,合并后103008行。</p>
从纯真数据库里获取ip列表
2011-05-19T00:00:00+08:00
monitor
perl
http://chenlinux.com/2011/05/19/get-iplist-from-qqwry
<p>首先申明只是一个简单的方式,因为打算的是提取总列表成bind9使用的acl格式,所以不在乎性能问题。<br />
第一步、从CZ88.NET下载QQWry数据库,然后运行IP.exe,选择“解压”,然后会在桌面生成一个qqwry.txt,这里就有四十多万行的ip记录。格式如下:<br />
起始ip 结束ip 大区域 小区域<br />
但是这个大区域也不是想像中的那么整齐,比如清华大学宿舍楼也是大区域的……<br />
好在我们DNS只需要一个大概的南北指向,根据电信占主流的现实,只要取出来联通的,其他都算电信就行了~<br />
第二步、把起始ip-结束ip改成acl需要的子网掩码格式,这一步用perl完成,全文如下:<br />
<code class="highlighter-rouge">perl
#!/usr/bin/perl -w
use Net::IPAddress::Util::Range;
while(<>){
next unless $_ =~ /^(\S+)\s+(\S+)\s+(.+)/;
my $range = Net::IPAddress::Util::Range->new({ lower => $1, upper => $2 });
map {printf "%s\t%s\n", $_, $3 } $range->tight()->as_cidrs();
}
</code><br />
其中tight()->as_cidrs()其实是Net::IPAddress::Util::Collection的函数(Range.pm里use了这个函数)。tight将不规律的ip段划分成规律的子网,cidrs将类似(1.1.1.0 .. .1.1.1.255)改成1.1.1.0/24。<br />
如果直接采用Net::IPAddress::Util::Range的$range->as_cidr()的话,它会把一个不规律的ip段取一个最近的规律子网来显示……比方1.59.0.0-1.60.149.255会被计算成1.57.0.0/13!!<br />
不过这个还有一个问题,就是没有多行合并,导致条目太多~~这个之后再看吧~</p>
链路故障应急处理脚本
2011-05-12T00:00:00+08:00
monitor
perl
http://chenlinux.com/2011/05/12/script-process-link-failure
<p>话接上篇,继续完成这个perl脚本。花了今天一天的时间,基本定稿如下:<br />
```perl#!/usr/bin/perl -w<br />
use Net::Ping::External qw(ping);<br />
use Tie::File;<br />
use Getopt::Long;</p>
<p>Getopt::Long::Configure (“bundling”);<br />
GetOptions(<br />
‘H:s’ => $ct_host, ‘host:s’ => $ct_host,<br />
‘T:i’ => $time, ‘time:i’ => $time,<br />
‘N:i’ => $fork_num, ‘number:i’ => $fork_num,<br />
‘h’ => $help, ‘help’ => $help,<br />
);</p>
<p>if( $help ) {<br />
print “Usage: ping_check.pl -H 10.168.168.251 -T 30 -N 10\n”;<br />
print “ -H/host: The switch ip address to be checked in china telecom;\n”;<br />
print “ -T/time: The seconds used for pinging;\n”;<br />
print “ -N/number: The number of fork processes to expect the remote hosts;\n”;<br />
print “ -h/help: The usage just you see now.\n”;<br />
exit 0;<br />
}</p>
<p>my $mark_file = ‘/tmp/mark_file’;<br />
my @last;</p>
<p>my $result = ping( hostname => “$ct_host”,<br />
count => $time * 5,<br />
size => ‘128’,<br />
timeout => ‘1’,<br />
#原生的Net::Ping模块需要自己while来控制sleep;<br />
#现在采用的Net::Ping::External是直接调用的外部ping命令,但默认也没有-i参数;<br />
#package中sub ping{}里把未定义的@_都给到了%args,<br />
#所以只需要在199行(即sub _ping_linux{}中)添加上-i $args{interval}就能用了。<br />
interval => ‘0.2’,<br />
);<br />
#用tie将数组@last锁定到文件上——我曾经想过直接锁个变量,但是似乎没有,只能数组或哈希?<br />
tie @last, ‘Tie::File’, $mark_file or die $!;</p>
<p>if ( $result && ($last[0] == 1) ) {<br />
print “ok\n”;<br />
}<br />
elsif ( $result && ($last[0] == 0) ) {<br />
print “Beginning recovery\n”;<br />
&parallel_manage(“recovery”, “$fork_num”);<br />
&sms_alarm(‘CNC recovery peer to intranet’);<br />
&email_alarm(‘CNC recovery peer to intranet’);<br />
} else {<br />
print “Error! Beginning change to template configuration\n”;<br />
&parallel_manage(“change”, “$fork_num”);<br />
&sms_alarm(‘CNC change peer to CT’);<br />
&email_alarm(‘CNC change peer to CT’);<br />
}</p>
<p>$last[0] = $result;<br />
untie @last;</p>
<p>sub email_alarm {<br />
use Net::SMTP_auth;<br />
my $email_message = shift;<br />
my $smtp = Net::SMTP_auth->new( Host => ‘smtp.domain.com’,<br />
Timeout => ‘30’,<br />
# Debug => ‘1’,<br />
);<br />
$smtp->auth(‘LOGIN’, ‘alarm@domain.com’, ‘password’);<br />
$smtp->mail(‘alarm@domain.com’);<br />
$smtp->to( ‘netadmin@domain.com’ );<br />
$smtp->data();<br />
$smtp->datasend(“To: Netadmin\@domain.com\n”);<br />
$smtp->datasend(“\n”);<br />
$smtp->datasend(“${email_message}\n”);<br />
$smtp->dataend();<br />
$smtp->quit;<br />
}</p>
<p>sub sms_alarm {<br />
use Net::MySQL;<br />
my $sms_message = shift;<br />
my %contacts = &get_contacts();<br />
my $mysql = Net::MySQL->new( hostname => ‘10.1.1.45’,<br />
database => ‘smsd’,<br />
user => ‘smsd’,<br />
password => ‘smsd’,<br />
);<br />
foreach my $send_number (values %contacts) {<br />
$mysql->query(<br />
“INSERT INTO outbox (number, text) VALUES ( $send_number, ‘$sms_message’)”<br />
);<br />
}<br />
$mysql->close;<br />
}</p>
<p>sub parallel_manage {<br />
#因为Expect模块本身使用了fork();要求运行在主进程中,所以在并发的时候不能采用多线程而得用多进程<br />
use Expect;<br />
use Parallel::ForkManager;</p>
<div class="highlighter-rouge"><pre class="highlight"><code>my $command = shift || 'id';
my $max_fork = shift || '10';
my @remote_list = ('10.1.1.64',
'10.1.1.50',
'10.1.1.35',
);
my $remote_host;
my %remote_result;
my $pm = Parallel::ForkManager->new( $max_fork, '/tmp/'); #采用Parallel::ForkManager模块时,如果需要子进程返回数据结果给父进程,必须把run_on_finish()放在fork之前 #Parallel::ForkManager模块实际上是采用文件存储的方式进行父子进程的数据通信,所以上面new的时候定义一个临时文件路径
$pm->run_on_finish (
sub { #主要有用的是子进程pid,子进程退出状态,返回数据的引用
my ($pid, $exit_code, $ident, $exit_signal, $core_dump, $reference) = @_;
if (defined($reference)) { #解引用后复制到数组并转存成哈希
my @data = @$reference;
$remote_result{"$data[0]"} = $data[1];
} else {
print qq|No message received from child process $pid!\n|;
}
}
);
foreach $remote_host (@remote_list) {
$pm->start and next;
my @check = ($remote_host, &ssh_expect($remote_host, $command)); #默认参数就是finish(0);需要返回数据时才加上引用
$pm->finish(0, \@check);
}
$pm->wait_all_children;
foreach my $key (sort keys %remote_result) {
print $remote_result{$key},"\n";
} }
</code></pre>
</div>
<p>sub ssh_expect {<br />
my ($host, $shell) = @_;<br />
my $exp = Expect->new;<br />
my $password = ‘password’;<br />
$exp = Expect->spawn(“ssh -l monitor -i /usr/local/monitor/conf/id_rsa -o ConnectTimeout=5 $host”);<br />
# $ENV{TERM}=”xterm”;<br />
# $exp->exp_internal(1);<br />
$exp->raw_pty(1);<br />
#关闭输出,不然expect会把整个session都print出来(实际是到STDERR)<br />
$exp->log_stdout(0);<br />
$exp->expect(2,[<br />
‘$’,<br />
sub {<br />
my $self = shift;<br />
$self->send(“su -\n”);<br />
}<br />
],<br />
[<br />
‘(yes/no)\?’,<br />
sub {<br />
my $self = shift;<br />
$self->send(“yes\n”);<br />
exp_continue;<br />
}<br />
]<br />
);</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$exp->expect(2, [
'Password:',
sub {
my $self = shift;
$self->send("${password}\n");
exp_continue;
}
],
[
'#',
sub {
my $self = shift;
$self->send("${shell}\n");
}
]
);
$exp->send("exit\n") if ($exp->expect(undef,'#')); #expect有before/match/after来返回相应的数据
my $read = $exp->before();
$exp->send("exit\n") if ($exp->expect(undef,'$'));
$exp->soft_close();
return $read; }
</code></pre>
</div>
<p>sub get_contact {<br />
open my FH, “/usr/local/nagios/etc/objects/contacts.cfg”;<br />
my %hash;<br />
local $/ = ‘define’;<br />
while(<fh>) {
next unless /pager\s+(\d{11})/;
my $sms_num = $1;
$hash{$1}=$sms_num if /email\s+([a-z\.]+?)\@bj.china.com/;
}
return %hash;
}
```</fh></p>
学习pm和bless的写法
2011-05-12T00:00:00+08:00
perl
http://chenlinux.com/2011/05/12/learning-bless-and-write-pm-demo
<p>考虑到公司环境必须先rsa_auth再su的问题,一般的pssh啊mussh啊sshbatch啊,都不能直接用,决定把上篇脚本里的相关部分抽出来成为一个模块,借机学习一下package和bless的简单概念:<br />
```perl<br />
#包名,如果做pm的话,必须和(.*).pm的名字一样<br />
package raocl;<br />
use Parallel::ForkManager;<br />
use Expect;<br />
#Exporter模块是perl提供的导入模块方法的工具<br />
use base ‘Exporter’;<br />
#Exporter有两个数组,@EXPORT里存的是模块的sub,@EXPORT_OK里存的是模块的var;<br />
#使用模块时只能调用这些数组里有定义的东西<br />
our @EXPORT = qw/new cluster/;<br />
#一般模块都有一个new方法来进行初始化定义<br />
sub new {<br />
#所有sub传入的第一个参数都是本身,所以要先shift出来,然后才是脚本显式传入的参数<br />
my $class = shift;<br />
#将参数转成哈希方式,并返回一个引用;<br />
#正规做法应该在这里指定一些必须要有的参数,比如passwd => %args{‘passwd’} || ‘123456’<br />
my $self = {@_};<br />
#bless上面返回的哈希引用到自己,再传递出去;以后在这之外的地方,使用被bless过的$self时自动就关联上new里的数据了。<br />
#这里我写的极简单,看比较正式的模块写发,这里对$class还要用ref();判断是不是引用等<br />
return bless $self,$class;<br />
}</p>
<p>sub cluster {<br />
#这里的$self就是上面被bless过的了<br />
my ($self, $command) = @<em>;<br />
my %remote_result;<br />
my $pm = Parallel::ForkManager->new( $self->{fork}, ‘/tmp/’);<br />
$pm->run_on_finish (<br />
sub {<br />
my ($pid, $exit_code, $ident, $exit_signal, $core_dump, $reference) = @</em>;<br />
if (defined($reference)) {<br />
my @data = @$reference;<br />
$remote_result{“$data[0]”} = $data[1];<br />
} else {<br />
print qq|No message received from child process $pid!\n|;<br />
}<br />
}<br />
);<br />
#直接使用bless过的$self解引用出来的hosts列表<br />
foreach my $remote_host (@{$self->{hosts}}) {<br />
$pm->start and next;<br />
#使用bless过的$self的sub完成expect功能<br />
my @check = ($remote_host, $self->pexpect($remote_host, $command));<br />
$pm->finish(0, \@check);<br />
}<br />
$pm->wait_all_children;</p>
<p>return %remote_result;<br />
}</p>
<p>sub pexpect {<br />
#还是同样的$self,然后才是上面调用时传递的$host和$shell<br />
my ($self, $host, $shell) = @_;<br />
#使用new里提供的passwd<br />
my $password = $self->{passwd};<br />
my $exp = Expect->new;<br />
$exp = Expect->spawn(“ssh -l admin -i /usr/local/admin/conf/id_rsa -o ConnectTimeout=5 $host”);<br />
$ENV{TERM}=”xterm”;<br />
$exp->raw_pty(1);<br />
#使用new里提供的开关<br />
$exp->exp_internal(“$self->{debug}”);<br />
$exp->log_stdout(“$self->{output}”);<br />
$exp->expect(2,[<br />
‘$’,<br />
sub {<br />
my $self = shift;<br />
$self->send(“su -\n”);<br />
}<br />
],<br />
[<br />
‘(yes/no)\?’,<br />
sub {<br />
my $self = shift;<br />
$self->send(“yes\n”);<br />
exp_continue;<br />
}<br />
]<br />
);</p>
<p>$exp->expect(2, [<br />
‘Password:’,<br />
sub {<br />
my $self = shift;<br />
$self->send(“${password}\n”);<br />
exp_continue;<br />
}<br />
],<br />
[<br />
‘#’,<br />
sub {<br />
my $self = shift;<br />
$self->send(“${shell}\n”);<br />
}<br />
]<br />
);<br />
$exp->send(“exit\n”) if ($exp->expect(undef,’#’));<br />
#因为shell命令执行的输出可能有滞后,所以将前后都输出<br />
my $read = $exp->before() . $exp->after();<br />
$read =~ s/[.+\@.+]//;<br />
$exp->send(“exit\n”) if ($exp->expect(undef,’$’));<br />
$exp->soft_close();<br />
return $read;<br />
}<br />
#package结尾,必须return一个1,原因未知……<br />
1;<code class="highlighter-rouge">
使用如下:
</code>perl#!/usr/bin/perl -w<br />
#可以在/usr/lib/perl5下,也可以在pl脚本的同目录下<br />
use raocl;<br />
#从文件中读取host列表为数组<br />
open FH, “./list”;<br />
my @hosts = <fh>;
#使用new初始化,传递host列表的引用给函数
$raocl=new raocl(hosts=>\@hosts,
fork => '10',
output => 0,
debug => 0,
passwd => '123456',);
%result = $raocl->cluster("$shell");
foreach my $host (keys %result) {
print $host."\t".$result{$key},"##############\n";
};```</fh></p>
perl的Expect模块
2011-05-09T00:00:00+08:00
perl
http://chenlinux.com/2011/05/09/expect-module-of-perl
<p>手头一批机器,因为历史的原因,有些密码登录、有些密钥登录,有些wheel组免密码su - root、有些又不行。为了统一管理操作,得想办法找一个能适应这四种情况的自动登录方法。</p>
<p>先看的Net::SSH2模块,用户、主机、密钥都支持列表,也有$ssh->exec();,但是最后这步su - root密码还是没法完成;<br />
然后看的Net::SSH::Expect模块,其实就是在Expect模块外面加一层shell调用ssh命令。没有独立的参数指定密钥等,而是写在ssh_option=>’ -i id_rsa ‘里,更糟糕的情况是:在无密码登陆时,只能使用$ssh->run_ssh();而不能用$ssh->login();——但关于初次登陆的(yes/no)?的问题,却只在login()里有处理,run_ssh()里没有!可以修改Net/SSH/Expect.pm文件,在sub run_ssh()的return之前添加相关处理的语句:<br />
<code class="highlighter-rouge">perl
$exp->expect(1,
[ qr/\(yes\/no\)\?\s*$/ => sub { $exp->send("yes\n"); exp_continue; } ],
);</code><br />
但运行的时候,一台内网机器,完成一次su -后ls的操作,居然平均需要消耗3s的时间。<br />
于是干脆使用原版的Expect模块,平均单次运行时间缩短到了1.3s,如下:<br />
<code class="highlighter-rouge">perl
#!/usr/bin/perl -w
use Expect;
#本来还打算用cgi模块改成web界面的,但运行时不时爆出“(70007)The timeout specified has expired:
#ap_content_length_filter: apr_bucket_read() failed”的error_log,
#百度谷歌的各种结果,如$|、STDOUT、Timeout、version都查了一遍,没有结果,只好暂时放弃
#use CGI::Simple;
#my $q = new CGI::Simple;
#my $host = $q->param('host');
#my $command = $q->param('command');
#print $q->header;
my $host = $ARGV[0];
my $command = $ARGV[1];
my $password = '1234!@#$';
my $exp = Expect->new;
$exp = Expect->spawn("ssh -l monitor -i /usr/local/monitor/etc/id_rsa $host");
#Expect模块的debug分析
#$exp->exp_internal(1);
$exp->expect(2, [
'\$',
sub {
my $self = shift;
$self->send("su -\n");
}
],
[
#凯哥三年前的博客复制的perldoc内容,都没有\号,实际必须有!
#perldoc说不加-re时就是完全匹配,这很容易让人理解为==的效果,但debug告诉我不是这样滴……
#Net::SSH::Expect里sub login()里也用了\号,见上。
'\(yes/no\)\?',
sub {
my $self = shift;
$self->send("yes\n");
exp_continue;
}
]
);
$exp->expect(2, [
'Password:',
sub {
my $self = shift;
#perldoc推荐使用$self->send_slow($timeout,"command\r");的方式,不过试了下,好慢啊,算了
$self->send("${password}\n");
exp_continue;
}
],
[
'#',
sub {
my $self = shift;
$self->send("${command}\n");
}
]
);
$exp->send("exit\n") if ($exp->expect(undef,'#'));
$exp->send("exit\n") if ($exp->expect(undef,'$'));
</code></p>
linux系统脚本中的awk一例
2011-05-06T00:00:00+08:00
bash
awk
http://chenlinux.com/2011/05/06/awk_variable_example_in_linux_system_script
<p>感谢@snowave童鞋,摘取了/usr/bin/run-parts最后一段的awk内容给我看:</p>
<table>
<tbody>
<tr>
<td>$i 2>&1</td>
<td>awk -v “progname=$i” ‘progname {print progname “:\n”;progname=””;} { print; }’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>上面这行其实相当于$i 2>&1</td>
<td>awk ‘BEGIN{print “‘$i’:\n”}{print}’ ,解释一下原版的用法:</td>
</tr>
</tbody>
</table>
<p>-v是定义一个awk内的变量,用来传递外面的shell变量到awk里;<br />
progname是就是如果存在这个变量执行下面语句段;<br />
然后在第一次print了之后把progname变量清空了,之后就不会再执行了。<br />
最后的print就是显示管道出来的结果了。</p>
<p>不过在我的理解和实验中,每次都经过一次if判断,执行效率远远不如begin处理。不知道为什么系统脚本会采用这种写法~~</p>
reiserfs和xfs的inode测试
2011-04-13T00:00:00+08:00
testing
xfs
reiserfs
http://chenlinux.com/2011/04/13/test-inode-support-of-reiserfs-xfs
<p>某应用的碎文件生成太多,大量消耗inode,不得不做迁移调整。<br />
原先的使用的NetApp存储,虽然可以调节inode,但是有一个上限,最高就是40000000——说实话真的挺小的啊~~<br />
一个同样大小的普通服务器ext3文件系统,在没有强行指定的情况下,inode都有将近3个亿,是netapp的7倍。<br />
了解了一下ext3文件系统在分区时指定inode最大可用数的情况,大致可以理解为小于可用空间大小/256。详细分析测试情况见:http://blog.wgzhao.com/2008/04/13/how-much-inodes-do-you-need.html,大意是inode个数由block大小和单个inode的字节决定。理论上最小block是1024,实际(采用-N强行指定的话)可能是256左右。</p>
<p>之前在做sersync的时候,发现开启xfs支持就可以对netapp使用inotify功能。因此可以认为netapp使用是(至少在inode上)类似xfs的文件系统。最终决定测试一个reiserfs和xfs在inode方面的情况——这两个文件系统都是非分布式的动态inode的文件系统。<br />
RHEL5默认是不支持这两个fs的。需要plus一下。方法见http://www.gnutoolbox.com/reiserfs-centos/和http://wiki.centos.org/AdditionalResources/Repositories/CentOSPlus的说明。一步一步照做即可。<br />
随后在vmware上添加一块10G的硬盘sdb,fdisk分区成sdb1和sdb2,分别mkfs.reiser和mkfs.xfs两个分区,挂载在/mnt1和/mnt2上。<br />
这个时候df -h和df -i查看如下:<br />
[root@localhost ~]# df -i<br />
Filesystem Inodes IUsed IFree IUse% Mounted on<br />
/dev/sda3 1240320 144694 1095626 12% /<br />
/dev/sda1 26104 40 26064 1% /boot<br />
/tmpfs 64314 1 64313 1% /dev/shm<br />
/dev/sdb1 0 0 0 - /mnt1<br />
/dev/sdb2 562240 2368 559872 1% /mnt2<br />
[root@localhost ~]# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/sda3 19G 4.8G 14G 27% /<br />
/dev/sda1 99M 17M 77M 18% /boot<br />
/tmpfs 252M 0 252M 0% /dev/shm<br />
/dev/sdb1 471M 33M 438M 7% /mnt1<br />
/dev/sdb2 545M 28M 517M 6% /mnt2<br />
另开窗口,分别运行for((i=1;i<562240;i++));do touch /mnt1/a_$i;done和for((i=1;i<562240;i++));do touch /mnt2/b_$i;done——为了加速,实际开了四个窗口,每个命令都是同时运行两个。<br />
一段时间后,mnt2的终端显示No space。于是中止。<br />
此时df -i和df -h的结果如下:<br />
[root@localhost ~]# df -i<br />
Filesystem Inodes IUsed IFree IUse% Mounted on<br />
/dev/sda3 1240320 144694 1095626 12% /<br />
/dev/sda1 26104 40 26064 1% /boot<br />
/tmpfs 64314 1 64313 1% /dev/shm<br />
/dev/sdb1 0 0 0 - /mnt1<br />
/dev/sdb2 124032 123397 635 100% /mnt2<br />
[root@localhost ~]# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/sda3 19G 4.8G 14G 27% /<br />
/dev/sda1 99M 17M 77M 18% /boot<br />
/tmpfs 252M 0 252M 0% /dev/shm<br />
/dev/sdb1 471M 60M 411M 13% /mnt1<br />
/dev/sdb2 545M 545M 144K 100% /mnt2<br />
可以发现,xfs的空间没有变化(说这句是因为ext3在强制inode数变大的时候,可用空间会变小),但inode总数“神奇”的缩水了4倍!!<br />
继续等待,直到mnt1的for循环执行完成,这时候df -h结果如下:[root@localhost ~]# df -h<br />
Filesystem Size Used Avail Use% Mounted on<br />
/dev/sda3 19G 4.8G 14G 27% /<br />
/dev/sda1 99M 17M 77M 18% /boot<br />
/tmpfs 252M 0 252M 0% /dev/shm<br />
/dev/sdb1 471M 134M 338M 29% /mnt<br />
根据最后的结果算471<em>1024/562240/2=0.43K, 545</em>1024/124032=4.5K。看来对于空文件,reiserfs很压缩,而xfs则老老实实的做block了。</p>
《SUSE Linux Enterprise Desktop System Analysis and Tuning Guide》读书笔记
2011-04-05T00:00:00+08:00
linux
http://chenlinux.com/2011/04/05/learning-suse-linux-enterprise-desktop-system-analysis-and-tuning-guide
<p>这是一本比上篇提到的老书新很多、厚很多的调优指南。虽然至今没用过suse,但同是linux内核,与redhat差距不算太大。目前只打印并看到了systemtap章节,感觉很多内容说的比上本书细致的多,继续做笔记~<br />
<!--more--><br />
1 General Notes on System Tuning<br />
1.2 Rule Out Common Problems<br />
检查/var/log/warn和/var/log/messages中的异常条目;<br />
用top或ps命令检查是否有进程吃了太多CPU和内存;<br />
通过/proc/net/dev检查网络问题;<br />
用smartmontools检查硬盘IO问题;<br />
确认后台进程都是在系统负载较低的时候运行的,可以通过nice命令调整其优先级;<br />
一台服务器运行太多使用相同资源的服务的话,考虑拆分;<br />
升级软件。<br />
2 System Monitoring Utilities<br />
2.1 Multi-Purpose Tools<br />
2.1.1 vmstat<br />
第一行输出显示的从最近一次重启以来的平均值<br />
各列说明:<br />
r 运行队列中的进程数。这些进程等待cpu的空闲以便执行。如果该数值长期大于cpu核数,说明cpu不足;<br />
b 等待除了cpu以外的其他资源的进程数。通常是IO不足;<br />
swpd 已用swap空间,单位KB;<br />
free 未用内存空间,单位KB;<br />
inact 可以回收的未用内存空间,只有当使用-a参数时才显示——建议使用该参数;<br />
active 使用中且没有回收的内存空间,同样只在-a时显示;<br />
buff 内存中的文件缓冲空间;相反,-a时不显示;<br />
cache 内存中的页缓存空间;-a不显示;<br />
si 每秒从内存移动到swap的数据大小,单位KB;<br />
so 每秒从swap移动到内存的数据大小,单位KB,以上两个数长期偏大的话,机器需要加内存;<br />
bi 每秒从块设备中获取的块数量——注意swap也是块设备,包含在内!<br />
bo 每秒发送到块设备的块数量,同样包括swap;<br />
in 每秒中断数,数值越大说明IO级别越高;<br />
cs 每秒文本交换数,这个代表内核从内存中某进程中提取替换掉另一个进程的可执行代码——茫然??<br />
us 用户空间的cpu使用率;<br />
sy 系统空间的cpu使用率;<br />
id cpu时间中的空闲比——就算它是0,也不一定就是什么坏事,还得看r和b两个数值来判断;<br />
wa 如果这个数不等于0,那说明系统吞吐在等待IO。这或许是不可避免的。比如如果一个文件是第一次被读取(即没有缓存),那同时的后台写必然挂起。这个数也是硬件瓶颈的一个指标(网络或者磁盘)。最后,还有可能是虚拟内存管理上的问题;<br />
st cpu用在虚拟管理上的比例。<br />
2.1.2 System Activity Information: sar and sadc<br />
sadc其实就是在/etc/cron.d中添加的任务。原始数据写入/var/log/sa/saDD中,报告数据写入/var/logs/sar/sarDD中。默认配置,数据每10分钟一收集;报告每6小时一收集。详见/etc/sysstat/sysstat.cron;数据收集脚本是/usr/lib64/sa/sa1;数据报告脚本是/usr/lib64/sa/sa2;有必要的话可以自己用这两个脚本收集性能数据。<br />
sar -f指定特定的数据文件出报告<br />
sar -P指定某一个CPU出报告<br />
sar -r显示内存信息:kbcommit和%commit显示了当前工作负载下可能需要的最大内存(含swap);<br />
sar -B显示内核页信息:majflt/s显示了每秒钟有多少页从硬盘(含swap)读入内存,这个数太大意味着系统很慢,而且内存不足;%vmeff显示了页扫描(pascand/s)及其相关的缓冲重用率(pgsteal/s),用以衡量页面回收的效率,数值接近100说明所有so的页都重用了,接近0说明没有被扫描的页,这都很好,但不要在0-30%之间。<br />
sar -d显示块设备信息,最好加上-p显示设备名;<br />
sar -n显示网络信息,包括DEV/EDEV/NFS/NFSD/SOCK/ALL<br />
2.2 System Information<br />
2.2.1 iostat<br />
-n显示nfs;<br />
-x显示增强型信息;<br />
2.2.3 pidstat<br />
-C “top”显示命令名中包括top字符串的目标。<br />
2.2.5 lsof<br />
无参数:打开的所有文件<br />
-i:网络文件<br />
2.2.6 udevadm<br />
本工具只有root可以使用<br />
2.3 Processes<br />
2.3.2 ps<br />
显示具体某进程:ps -p $(pidof ssh)<br />
显示格式和排序:ps ax –format pid,rss,cmd –sort rss<br />
显示单独进程:ps axo pid,$cpu,rss,vsz,args,wchan,etime<br />
显示进程树:ps axfo pid,args<br />
2.3.4 top<br />
默认每2秒钟一刷新;<br />
显示一次即退出:-n 1<br />
shift+p——以CPU使用率排列(默认);<br />
shift+m——以常驻内存排列;<br />
shift+n——以进程号排列;<br />
shift+t——以时间排列;<br />
2.4 Memory<br />
2.4.1 free<br />
free -d 1.5——每1.5秒一刷新数据<br />
2.4.3 smaps<br />
在/proc/${pid}/smaps中看到的是进程当前的内存页数量,即除掉共享内存以外的真正进程使用的内存大小。<br />
2.5 Networking<br />
2.5.1 netstat<br />
-r路由;-i网卡;-M伪装链接;-g广播成员;-s信息<br />
2.5.2 iptraf<br />
iptraf -i eth0 -t 1 -B -L iptraf.log<br />
eth0网卡一分钟内的信息,后台收集,记入iptraf.log中。<br />
2.6 The /proc File System<br />
/proc/devices 可用设备<br />
/proc/modules 已加载内核模块<br />
/proc/cmdline 内核命令行<br />
/proc/meminfo 内存使用详细信息<br />
/proc/config.gz 内核当前运行配置的压缩文件<br />
详细说明见/usr/src/linux/Documentation/filesystems/proc.txt<br />
执行的进程和库文件以及他们在内存的地址信息见/proc/***/maps文件。</p>
《Tuning Red Hat Enterprise Linux on IBM server xSeries Servers》读书笔记
2011-04-01T00:00:00+08:00
linux
http://chenlinux.com/2011/04/01/learning-tuning-red-hat-enterprise-linux-on-ibm-server-xseries-servers
<p>一本很老的书,还是RHEL3时代的,在陪GF的空隙一点点读完,把笔记整理一下发在这里,只包括自己不知道或者说容易忘记的内容,不代表调优指南。<!--more--><br />
1 Tuning the operating system<br />
1.1 Disabling daemons<br />
关闭不必要的后台进程。RHEL3中,默认启动的后台进程有:<br />
apmd 高级电源管理<br />
autofs 自动挂载<br />
cups 通用UNIX打印机系统<br />
hpoj 惠普打印机支持<br />
isdn 调制解调器<br />
netfs nfslock portmap NFS支持<br />
pcmcia PCMCIA支持<br />
rhnsd 自动升级<br />
sendmail 邮件转发程序<br />
xfs 桌面程序<br />
1.2 Shutting down the GUI<br />
runlevel:<br />
0 halt立刻关机immediately shut down<br />
1 single单人<br />
2 multi-user without NFS(这个说明和一般的说法不太一样~)<br />
3 full multi-user<br />
5 X11<br />
6 reboot<br />
修改/etc/inittab如下:<br />
id:3:initdefault: #runlevel<br />
#4:2345:respawn:/sbin/mingetty tty4 #关闭多余控制台<br />
注意:留3个,以免在被攻击的时候自己反而进不去了!<br />
1.4 Changing kernel parameters<br />
/proc/loadavg 系统负载1/5/15分钟<br />
/proc/stat 内核状态:进程/swap/磁盘IO<br />
/proc/cpuinfo CPU信息<br />
/proc/meminfoo 内存信息<br />
/proc/sys/fs/* linux可用文件数及磁盘配额<br />
/proc/sys/kernel/* 进程号范围/系统日志级别<br />
/proc/sys/net/* 网络细节<br />
/proc/sys/vm/* 内存缓冲管理<br />
1.7 Tuning the processor subsystem<br />
CPU超线程注意事项:<br />
注意使用SMP的kernel<br />
实际CPU数越多,超线程意义越小:<br />
2核:提升15-25%<br />
4核:提升1-1%<br />
8核:提升0-5%<br />
1.8 Tuning the memory subsystem<br />
如果决定调整/proc/sys/vm/*的参数,最好一次只调整一个。<br />
vm.dbflush前3个参数分别为:<br />
nfract 在buffer被转存到disk前允许的最大buffer比率<br />
ndirty 将buffer转到disk时一次允许操作最大的buffer数<br />
nfract_sync 转存时允许buffer中dirty数据的最大比率<br />
vm.kswapd<br />
tries_base 一次swap传输时的pages数。如果swapping较大,适当增加该值<br />
tries_min kswapd运行时交换的pages的最小数<br />
swap_cluster kswapd一次写入pages的数。太小会增加IO次数,太大又要等待请求队列<br />
1.9 Tuning the file subsystem<br />
磁盘访问速度是ms级别的,而内存是ns,PCI是us。<br />
磁盘IO是最关键的问题服务器举例:<br />
文件/打印服务器:所有数据从磁盘读取<br />
数据库服务器:大量IO,在内存和磁盘间交换数据<br />
磁盘IO不是最关键的问题服务器举例:<br />
邮件服务器:网络状况才是最关键的。<br />
web服务器:网络和内存才是最关键的…..<br />
1.9.5 The swap partition<br />
创建多个swap区有助于提升swap性能<br />
通常情况,多个swap采用顺序读写,即只有/etc/fstab中排名在前的swap区耗尽的情况下,才会使用下一个swap区;<br />
可以在fstab中定义优先级,类似”/dev/sda2 swap swap sw,pri=5 0 0”的格式;<br />
相同优先级的swap区,系统会并发使用,不同优先级之间依然要等待耗尽!——另外,如果相同优先级的swap区有一个性能较差,会连带影响整个swap性能。<br />
1.10 Tuning the network subsystem<br />
网络问题经常会导致其他伴生问题。比如:块大小太小会给CPU利用率带来显著影响;TCP连接数过多会带来内存使用率的急速上升……<br />
经常被打开的net.ipv4.tcp_tw_reuse和net.ipv4.tcp_tw_recycle的作用:缓存TCP交互中的客户端信息,包括交互时间、最大段大小,阻塞窗口。详见RFC1644。<br />
net.ipv4.tcp_fin_timeout可以缩短TCP建连时最后发送FIN序列的时间,以便快速释放内存提供给新进连接请求。但是修改这个的时候也要谨慎,因为由此导致的死套接字数量可能引起内存溢出!<br />
net.core.wmem_max/net.core.rmem_max定义在每个TCP套接字创建时划分的内存大小,推荐设置8MB。<br />
net.ipv4.tcp_wmem/net.ipv4.tcp_rmem的最后一个数字不能大于上面core的定义。<br />
net.ipv4.tcp_max_syn_backlog队列存放半连接。这些连接可能是因为客户端的连接异常,也可能仅仅是因为服务器负载太高导致。除了半连接,这个配置对防范拒绝服务攻击也有效。<br />
net.ipv4.ipfrag_low_thresh/net.ipv4.ipfrag_high_thresh规范ip碎片,一旦触底,内核会开始丢包。这对于NFS和samba等文件服务器很重要,建议设置为256和384MB。<br />
2 Tuning tools<br />
2.3 top<br />
STAT:S=SLEEPING,R=RUNNING,T=TRACED/STOPPED,D=INTERRUPTIBLE SLEEP,Z=ZOMBIE<br />
2.3.1 Process priority and nice levels<br />
优先级从19(最低)到-19(最高),默认是0。启动进程时指定nice -n 19 command,启动后改变renice 19 command<br />
2.4 iostat<br />
tps:transfers per second,多个单独的IO请求,可以组合在一次transfer请求中。<br />
Blk_read/s,Blk_wrtn/s:每秒的读写块个数。block大小和transfer大小一样各不相同。一般是1、2、4KB,采用如下命令查看:dumpe2fs -h /dev/sda1 | grep -F ‘Block Size’<br />
2.5 vmstat<br />
Process:r:等待运行的进程数,b:不可中断睡眠中的进程数<br />
Swap:单位是KBps<br />
CPU:us:非内核时间,包括user和nice,id:在linux2.5.41前,这个数值包括了IOwait时间在内……<br />
2.11 ulimit<br />
-H和-S分别是hard和soft,开机启动指定的话,修改/etc/security/limits.conf即可。<br />
2.12 mpstat<br />
用来在多CPU的机器上查看每个CPU的情况。<br />
3. Analyzing performance bottlenecks<br />
3.1 Identifying bottlenecks<br />
快速调优策略:<br />
a. 了解你的系统;<br />
b. 备份系统;<br />
c. 监控、分析系统性能;<br />
d. 缩小瓶颈,找出根源;<br />
e. 解决瓶颈的时候一次只修改一个地方;<br />
f. 返回c步骤继续,直到满意。<br />
3.1.1 Gathering information<br />
在收到“服务器出问题了”的报警时,提出下列问题,可以更加有效地收集信息进行故障定位:<br />
Q:服务器的完整描述?包括:模块、使用时长、配置、外围设备、操作系统版本号……<br />
Q:能准确描述一下问题所在么?包括:症状表现、各种错误日志记录……<br />
Q:问题是谁碰到/发现的?一个人、某些特定的人群,还是所有的用户?由此可以大概猜测问题是网络、应用还是客户电脑。另外:性能问题可能不会立刻从服务器反应到客户端上来,因为网络延迟经常会覆盖掉其他问题。这个延迟包括网络设备,也包括其他服务器提供的网络服务,比如域名解析~<br />
Q:问题可以再现么?所有可以再现的问题都是可以解决的!<br />
重现故障的步骤是什么?这可以协助你在测试环境完成调优工作。<br />
问题是持续发生的么?如果是断续发生的,赶紧找出让它重现的办法,最好就是能按你的剧本指令重现……<br />
问题是不是周期性的固定某个时间发生?查查那时候是不是有人登陆了?尝试梗概系统,看问题会重现么?<br />
问题真的很不常见?如果真是如此,那只能说rp有问题了,事实上,绝大多数问题都是可重现的~对没法重现的,那就出网管绝招:reboot、然后更新升级驱动和补丁。<br />
Q:问题什么时候开始的?逐渐显现还是突然爆发?如果是逐渐,那应该是积累出来的;如果是突然,那考虑是不是外设做了改动。<br />
Q:服务器是不是有变动,或者客户端的使用方法变了?<br />
Q:事情紧急么?要求几分钟内搞定还是未来几天?<br />
3.1.2 Analyzing the server’s performance<br />
在任何排障动作前,牢记备份!!<br />
有必要为服务器创建一份性能日志,内容包括:进程、系统、工作队列、内存、交换页、磁盘、重定向、网卡……<br />
3.2 CPU bottlenecks<br />
动态应用/数据库服务器,CPU常常是瓶颈,但实际经常是CPU在等待其他方面的响应。<br />
3.2.1 Finding CPU boottlenecks<br />
注意:同时不要运行多个工具,以免给CPU增加负载<br />
3.2.2 SMP<br />
进程在CPU之间进行切换时需要消耗一点的时间,所以绑定CPU比较有用。<br />
3.2.3 Performance tuning options<br />
关闭非必须进程;调成优先级;绑定CPU;CPU主频,是否多核;更新驱动;<br />
3.3 Memory bottlenecks<br />
free命令参数-l -t -o,分别表示low/high,total,old(不显示buffer信息)<br />
3.3.2 Performance tuning options<br />
调整页大小,默认是4/8KB;限定user资源limits.conf;……<br />
3.4 Disk bottlenecks<br />
常见问题:一、硬盘数太少;二、分区数太多,导致磁头寻址时间变大。<br />
3.4.1 Finding disk bottlenecks<br />
写缓冲;磁盘控制器负载;网络延时导致响应慢;IO等待队列<br />
随机读写还是顺序读写?单次IO大还是小?<br />
表3-2<br />
磁盘转速 latency seek-time random-access-time IOPS Throughout<br />
15000 2ms 3.8ms 6.8ms 147 1.15MBps<br />
10000 3ms 4.9ms 8.9ms 112 900KBps<br />
7200 4.2ms 9ms 13.2ms 75 600KBps<br />
在一个大概70%读30%写的随机IO型正常负载的服务器上,采用RAID10比RAID5能提高50-60%的性能。<br />
打开文件太多时,会因为寻址时间太长导致响应的变慢<br />
iostat的指标:<br />
%util 被IO请求消耗的CPU比例<br />
svctm 完成一个请求的平均时间,单位ms<br />
await 一个IO请求等待服务的平均时间,单位ms<br />
avgqu-sz 平均队列长度<br />
avgrq-sz 平均请求大小<br />
rrqm/s 发送到磁盘的每秒合并读请求数<br />
wrqm/s 发送到磁盘的每秒合并写请求数<br />
3.4.2 Performance tuning options<br />
顺序读写换磁头;随机读写加磁盘;用硬件RAID卡;加内存<br />
3.5 Network bottlenecks<br />
3.5.2 Performance tuning options<br />
检查路由配置;子网;网卡速率;TCP内核参数;换网卡;bonding<br />
4 Tuning Apache<br />
4.1 Gathering a baseline<br />
吞吐量:每秒请求数和每秒传输字节数;请求处理响应时间……<br />
4.5 Operating system optimization<br />
文件打开数;进程数;文件访问时间——不记录atime的作用是消减IO峰值!<br />
4.6 Apache 2 optimizations<br />
如果文件是通过NFS方式发布的,apache不会采用sendfile方式缓存文件,配置文件请选择“EnableSendfile Off”!<br />
4.6.1 Multi-processing module directives<br />
经常需要重启的,加大StartServer;<br />
负载较大的,加大MinSpareServers到25,MaxSpareServers到125;<br />
MaxClients最大只能是256,内存不足时应该减少;<br />
4.6.2 Compression of data<br />
默认的6级压缩比,可以带来72%的带宽减小。太高级别压缩,对CPU有影响。<br />
在测试中,启用压缩的apache带宽减小70%;cpu负载上升87%到饱和状态,能同时处理的客户端请求数降到三分之一。<br />
vary头的作用:告知代理服务器对支持压缩的客户端只发送压缩后的内容。<br />
apache2只在客户端请求包含Accept-encoding: gzip和Accept-encoding: gzip, deflate的时候才压缩数据。<br />
4.6.3 Logging<br />
使用WebBench的时候,一般会有2%的请求是404的,这可能导致error_log迅速变大!<br />
5 Tuning database servers<br />
5.1 Important subsystems<br />
CPU:<br />
数据库都是多线程的,最好使用16核以上CPU,2级缓存相当重要,命中率最好在90%以上;<br />
内存:<br />
缓冲是数据库最重要的部分。编译内核时请确认CONFIG_HIGHMEM64G_HIGHPTE=y这项。<br />
磁盘:<br />
数据库会有大量的磁盘IO以完成数据在内存和硬盘的交换。一般每个xeon的CPU需要对应10块高速硬盘,最好能有50块10000转的磁盘。IBM的xSeries 370使用450块10000转磁盘以达到最大吞吐量——每分钟40000次交换。</p>
<p>笔记到此为止。之后的内容是DB2的调优,samba、ldap、lotus章节,就没看了……</p>
php编译参数问题一例
2011-03-31T00:00:00+08:00
php
http://chenlinux.com/2011/03/31/problem-of-php-compile-option
<p>某php应用在给图片加水印的时候,显示的中文全都成了乱码,而开发同事在它本机(ubuntu)上apt安装的lamp上显示没有问题。仔细检查过了从env到encoding到phpinfo,都没有发现问题——都符合GD函数imagettftext()的utf8要求。<br />
还好有google,发现一个类似的文章,提出是php的编译参数中有一个–enable-gd-jis-conv,会把ttf字库中非标准拉丁文的部分,按照日文顺序映射,imagettftext()的默认编码其实被隐形指定成了日文编码euc-jp,中文自然就不正常了!<br />
然后赶紧重新看phpinfo,真有这个参数。重新编译php,随后恢复正常显示了。<br />
编译参数还真是不能大意啊~~</p>
awk单行命令
2011-03-28T00:00:00+08:00
bash
awk
http://chenlinux.com/2011/03/28/using-awk-as-one-line-command
<p>一眨眼又几天没更新,中午在Q群里聊到一个单行命令,随手记录下。<br />
A需求:某文件如下<br />
aaa<br />
bbb<br />
aaa<br />
aaa<br />
ccc<br />
aaa<br />
ddd<br />
……<br />
通过命令改成如下格式:<br />
1<br />
bbb<br />
3<br />
5<br />
ccc<br />
7<br />
ddd<br />
……<br />
方法如下:<br />
<code class="highlighter-rouge">bash
echo -en 'aaa\nbbb\naaa\naaa\ndddd'|awk 'BEGIN{tag=1}{if(/aaa/){print tag;tag+=2}else{print}}'</code><br />
或者更短一点:<br />
<code class="highlighter-rouge">bash
echo -en 'aaa\nbbb\naaa\naaa\ndddd'|awk 'BEGIN{tag=1}{if(/aaa/){$0=tag;tag+=2};print}'</code><br />
B需求:<br />
aaa不是文件一行的全部内容,而只是一部分。<br />
方法如下:<br />
<code class="highlighter-rouge">bash
echo -en 'aaa\nbbb\naaa\nfdaaafdaaa\ndddd'|awk 'BEGIN{tag=1}{gsub(/aaa/,tag) && tag+=2;print}'</code><br />
以上三种,凯的perl版本分别如下:<br />
<code class="highlighter-rouge">perl
perl -nale 'BEGIN{$tag = 1}if(/aaa/){print $tag;$tag+=2}else{print}'
perl -nalpe 'BEGIN{$tag = 1};$_=$tag and $tag+=2 if /aaa/'
perl -nalpe 'BEGIN{$tag = 1};$tag+=2 if s/aaa/$tag/'</code></p>
BSD上的流量监控脚本
2011-03-21T00:00:00+08:00
monitor
FreeBSD
awk
http://chenlinux.com/2011/03/21/bsd_flow_monitor_by_awk
<p>之前有过一篇linux上的流量监控脚本的博文,是利用procfs进行数据运算。但BSD上的procfs和linux有所不同,它只包含了进程的信息,没有系统的统计。所以只能通过其他方法。<br />
linux上另一种获取网卡总流量的方法是ifconfig命令,这个命令其实也是读取proc;但是BSD上的ifconfig输出里也没有……<br />
不过BSD上倒不是没法查流量上,事实上另外有两个命令,在实时观测中更加好用,一个是systat -if 1,几乎就是一个无色版的iptraf;另一个是netstat -idbhI bce0 1。<br />
解释一下,systat是bsd上用来查看系统信息的一个超级利器,-if是-ifstat的简写,类似还有-vmstat等等。netstat是专门用来显示网络状态的,最常用的就是-ant显示所有的TCP链接,这里用的idbhI,表示interface、drop、bytes、human,也就是用方便读取的格式输出某网卡的流量值。<br />
但是这两个命令,都是持续输出,必须接到^C信号才会退出运行。在实时管理时很好用,在做监控脚本的时候,就弄巧成拙了……<br />
好在netstat参数多,调整一下,使用idbnf参数(family)即可输出网卡总流量值,然后按照linux上一样的思路进行计算了。<br />
最终脚本如下:<br />
<code class="highlighter-rouge">bash#!/bin/bash
/usr/local/bin/gawk 'BEGIN{flow="netstat -idbnf inet";while((flow) | getline){now_in[$1]=$7;now_out[$1]=$10};time=systime()}{if_in[$1]=(now_in[$1]-$2)*8/(time-$4);if_out[$1]=(now_out[$1]-$3)*8/(time-$4)}END{printf "OK. The flow is %.2f,%.2f,%.2f,%.2f Kbps | bce0_in=%d;0;0;0;0 bce0_out=%d;0;0;0;0 bce1_in=%d;0;0;0;0 bce1_out=%d;0;0;0;0",if_in["bce0"]/1024,if_out["bce0"]/1024,if_in["bce1"]/1024,if_out["bce1"]/1024,if_in["bce0"],if_out["bce0"],if_in["bce1"],if_out["bce1"];for(i in now_in){print i,now_in[i],now_out[i],time > "/tmp/if_flow.txt"}}' /tmp/if_flow.txt
</code><br />
脚本就一行,不过就有一个缺点比不上分开写很多行的shell脚本,第一次运行前必须手动touch /tmp/if_flow.txt。因为如果这个文件不存在的话,awk会报错,执行不到END{}来,不会自动生成这个文件的……<br />
另:systime()函数是gawk特有的,而BSD上默认的是awk,所以需要安装gawk(在/usr/ports/lang/gawk目录下make&&make install);或者在awk中采用shell变量,定义time=’<code class="highlighter-rouge">date +%s</code>‘来调用了。</p>
gearman单机试验
2011-03-18T00:00:00+08:00
perl
gearman
http://chenlinux.com/2011/03/18/gearmand-test-on-single-server
<p>想把前端缓存几十台服务器的访问日志数据计入数据库中,以便核算各频道带宽。按照一般流量统计的惯例,在服务器上设定crontab每5分钟rotate一次access.log。但是想到一个问题——当A服务器select了数据库里的数据但还没来得及update时,B服务器也开始执行任务select数据来了,最后的结果就不准确了——我相信这个问题对coder来说很低级,不过我是op,搞不来……</p>
<p>不过op有op的思路——咱把所有的数据汇总由一台服务器来写数据库。考虑scp或者rsync可能会出现的各种未知失败(这个失败太普遍了,相信所有的op都有体会),打算试试gearman。</p>
<p>gearman出现的本意,是由jobserver来派发client的jobs到workers完成,比如张宴提到金山用它来分发上传图片的缩略裁剪(这也正是LiveJourna发明gearman的初始用途);比如TimYang提到用它来分发监控任务;OSCON2009的文档上,还提到了搜索引擎、分布式文件系统、map/reduce、日志分析、mysql集群处理等多种应用。</p>
<p>其中关于日志分析的结构图如下:</p>
<p><img src="/images/uploads/gearman4log.jpg" alt="gearman" /></p>
<p>好了。上面都是虚的。现在开始做实验。</p>
<p>下载gearman的c语言版,毕竟单纯就为了记录下带宽值的话,没必要下perl版的来折腾——注意,和memcached一样,gearman也采用了libevent,所以必须先安好libevent:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget http://launchpad.net/gearmand/trunk/0.14/+download/gearmand-0.14.tar.gz
tar zxvf gearmand-0.14
<span class="nb">cd</span> !<span class="err">$</span>
./configure <span class="o">&&</span> make <span class="o">&&</span> make install
</code></pre>
</div>
<p>默认会采用sqlite存储持久化队列。如果觉得memcached什么的更有爱,也可以–with。</p>
<p>安装完成后,会在/usr/local/bin下生成gearman和gearmand两个文件,前一个是worker和client共用的;后一个是jobserver。</p>
<p>首先要启动jobserver:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>gearmand -d -p 7003
</code></pre>
</div>
<p>然后开启一个client:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>echo 'test' | gearman -h 127.0.0.1 -p 7003 -f testwork
</code></pre>
</div>
<p>资料都说7003是gearman的默认端口,但C版的必须明确定义才行。</p>
<p>这个时候,可以telnet 127.0.0.1 7003上去,输入status看看了,上面显示“1 0 0”,表示有一个请求在队列中等待执行了。</p>
<p>然后注册一个worker:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>gearman -n -w -f testwork -- ls -lh
</code></pre>
</div>
<p>资料也都没有要使用-n参数。但如果不用这个,worker会在接收执行完队列中的第一个任务后就主动退出了!</p>
<p>ok,现在执行结果出来了。在client端(因为单机执行,其实就是运行client命令的ssh窗口),显示出来了一串文件信息,就是worker端的ls -lh结果。</p>
<p>然后重新注册worker,以测试client传递的内容能被worker正确处理:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>gearman -n -w -f testwork | awk '{print $0}'
</code></pre>
</div>
<p>嗯?worker端还是没有反应啊?切到client来,发现client这边显示了test!难道是标准输出就是用来返回给client的?那试试别的命令试试吧。</p>
<p>重新注册worker:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>gearman -n -w -f testwork | awk '{system("touch /tmp/"$0}'
</code></pre>
</div>
<p>这时候也可以telnet上去看看status,会显示“0 0 1”,表示有一个注册在jobserver上的worker;</p>
<p>然后重新发起client:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>echo 'test' | gearman -h 127.0.0.1 -p 7003 -f testwork
</code></pre>
</div>
<p>在到/tmp下一看,确实出现了/tmp/test文件了~~</p>
<p>不过,如何确定这个/tmp/test是worker生成的,而不是client呢(单机测试就是郁闷啊……)</p>
<p>修改一下worker:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>gearman -n -w -f testwork | awk '{system("sleep 100;touch /tmp/"$0}'
</code></pre>
</div>
<p>重新发起client,因为有sleep 100在,这次client没有立刻退出,但为了测试需要,按下Ctrl+C,终止client的运行,以确保命令不会是由client执行的(更确保一点,可以退出ssh会话)。</p>
<p>切换到/tmp/目录下,用stat命令查看/tmp/test文件的CTIME——目前还是上一次试验时的生成时间。</p>
<p>这时候telnet看status,显示是“1 0 1”,表示一个job正在运行。</p>
<p>再过一会儿,stat看,发现/tmp/test的CTIME突然变成一个新的时间了。可见touch命令确实是由worker执行的。</p>
linux小报错一例
2011-03-17T00:00:00+08:00
linux
http://chenlinux.com/2011/03/17/initctl-error
<p>需要给某台服务器加内存,准备关机的时候,却一直报错。考虑直接断电危害比较大,还是找找原因,报错如下:<br />
shutdown: timeout opening/writing control channel /dev/initctl <br />
init: timeout opening/writing control channel /dev/initctl <br />
从messages日志里没有发现任何附加信息。只能求助百度。好在解答很多:<br />
是原有的initscripts和SysVinit找不到了导致的。<br />
传上去initscripts-8.45.19.EL-1.x86_64.rpm和SysVinit-2.86-14.x86_64.rpm两个包,rpm -ivh安装。<br />
重新poweroff还是报错;但强制-f跳过shutdown过程后,成功了。稍后再启动设备,试着再敲一次poweroff,这次就不报错,成功关机了。</p>
squid/varnish/ats简单测试
2011-03-15T00:00:00+08:00
CDN
squid
varnish
ats
http://chenlinux.com/2011/03/15/simple-test-between-squid-varnish-ats
<p>简单的试试varnish和apache traffic server。跟squid进行一下对比。<br />
介于varnish和ats都是出现不太久的东西(至少是开源流行不长),选择其最新开发版本测试,正巧都是2.15版。呵呵~<br />
varnish配置文件,只修改了backend到具体某台nginx发布点上。其他都是反向代理缓存的标配。(可参考张宴博客http://blog.s135.com和varnish权威指南http://linuxguest.blog.51cto.com/195664/354889)<br />
ats说明更少,从目前的资料看,修改了records.config中的监听,cache.config遵循源站未改,remap.config添加了map一条,绑定域名到同一台nginx上。<br />
注:varnish和ats都没有修改缓存路径,即分别为var/varnish/varnish.cache和var/trafficserver,都是磁盘。</p>
<p>然后从线上某台squid2.7上收集www.domain.com下的html页面url一共164条,存成urllist。<br />
使用http_load进行测试。第一遍,先用http_load -r 100 -s 10完成cache的加载;第二遍改用-r 1000 -s 60进行测试。<br />
1、先是varnish<br />
开另一个窗口看netstat,发现在第一次加载的时候,varnish启动了相当多的链接到后端nginx请求数据!第二遍时,-r1000一直在刷wrong,修改成-r900就没有问题。最后的报告显示fetch/sec还略大于指定的900达到990,建连时间平均1.3ms,响应时间1.8ms。<br />
2、然后ats<br />
-r1000也报wrong,于是同样使用-r900,fetch/sec和和建连时间与varnish相近,响应时间2.1ms。<br />
从trafficserver_shell的show:proxy_stats和show:cache_stats命令结果来看,缓存命中率98%,磁盘IO几乎没有。可见其实都在内存中返回了。<br />
3、最后squid2.7.9<br />
-r900时,fetch/sec只有880,响应时间1.9ms;提高到-r1000时,没有wrong报错,fetch/sec下降到850,响应时间2.3ms;另一个窗口的netstat命令直接卡住……<br />
squid按照公司默认做法,缓存目录建在了tmpfs上。从squidclient来看,98%的命中率中只有三分之一是直接通过cache_mem返回的,另三分之二是通过cache_dir的tmpfs返回。</p>
<p>另:最后du -sh查看三者的缓存目录大小,赫然发现squid的是19M,ats是39M,varnish是41M。这个差别也是比较怪异的,值得后续研究……</p>
<p>从这个简单测试结果看,squid的稳定性依然没的说:对于大多数情况来说,是乐于见到这种宁愿响应慢点点也要保证响应正确的情况的;varnish在大批量回源时对后端服务器的冲击,显然比较让人担心;ats和varnish具有同样高效的响应速度(和高压下的错误……),而且其详细到甚至稍显繁琐的那堆config文件的配置格式,相比varnish来说,更加贴近运维人员(也就是说看起来不像编程语言)~~</p>
Facebook运维工程师的一天
2011-03-03T00:00:00+08:00
http://chenlinux.com/2011/03/03/zz-a-day-in-the-life-of-facebook-operations
<p>原文地址:http://linuxsysadminblog.com/2010/09/a-day-in-the-life-of-facebook-operations/<br />
顺便说一句,这个linuxsysadminblog.com确实不错。<br />
文章是笔者在现场听Facebook系统工程师汤姆·库克在2010年的surge可扩展和性能大会上的讲演时记录下来的——<br />
这是到目前为止最火爆的讲座了,还没开始就只剩下点站着的空间……<br />
Facebook运维需要支持多大的应用环境:<br />
1、在facebook花上每个月7亿分钟(不太明白这个意思,一个月明明只有43200分钟)<br />
2、60亿次内容更新;<br />
3、30亿张图片;<br />
4、实现100万连接;<br />
5、5亿活跃用户。<br />
基础设施建设的发展:<br />
1、租用IDC达到瓶颈;<br />
2、开始自建;<br />
3、目前已为加利福尼亚和弗吉尼亚两州服务。<br />
发展历程:<br />
从LAMP起步,然后拆分负载均衡,web服务器,app服务器,memcached,数据库。<br />
最早的Facebook是一个单纯的发布在apache上的php站点。但后来php不足以支持Facebook的访问了,现在,Facebook已经开始编译一些php功能成C++程序,这就是业界闻名的HipHop。<br />
Facebook大概拥有世上最大的memcached集群,超过300TB的数据存储在memcached内存中。<br />
使用flashcache来改善mysql性能。<br />
已实现支持的服务:<br />
新闻feed、搜索、缓存。<br />
服务使用中的编程语言:<br />
C++、php(前台)、python、ruby、java、erlang(聊天室)<br />
各种编程语言间如何交互数据?json?soap?都不是。Facebook专用一个为各编程语言开发服务的软件框架。所有的Facebook系统后面,都是统一的一个平台。<br />
Facebook每天都在担心:<br />
开发、监控、数据管理、代码上线。<br />
Facebook使用centos操作系统!<br />
系统管理:<br />
配置管理;<br />
系统管理——用CFengine;<br />
按需管理。<br />
开发:<br />
前台部分——每天都有新代码上线。代码统一协调,所有人都在IRC的频道里交流。每个人都可以知道发生了什么,而不单单是工程师自己。<br />
1、程序推送按需分发;<br />
2、代码分发通过BT的方式;<br />
3、php是经过编译的,数百MB的二进制文件通过极小的BT种子迅速下发;<br />
4、完成全网更新只需要1分钟。<br />
后台部分——只有开发和运维。开发工程师写代码、测试、演示。<br />
1、这样能迅速得到性能数据;<br />
2、揭露各环节的真实交互(这里翻译的应该不对,看不懂……);<br />
3、没有所谓的“提交、退出”;<br />
4、全面参与应用变成产品的过程;<br />
5、运维被“嵌入”每个开发团队。<br />
代码的每一处修改和推送,都必须详细记录。<br />
Facebook的服务器性能指标在线监控<br />
ganglia监控系统:快速、便捷、超过500万的监控指标、可以通过网格和池的方式进行规划。<br />
自主的监控系统<br />
nagios监控系统:用来给各团队发报警,最开始是用的email。<br />
Scribe高性能日志系统:最开始用的是syslogd,同时在用hadoop和hive。<br />
怎么运行的呢:<br />
1、定义要明确的依赖关系;<br />
2、固定的失败次数;<br />
3、服务器是第一步,系统架构设计。<br />
4、现在重点是搞集群。通过功能不同进行逻辑关系划分(web、db、feed等)<br />
5、下一步是数据中心。尤其是灾备。<br />
6、不断的沟通——信息永远在共享中。<br />
7、IRC。<br />
8、大量的自动化数据获取和设置。<br />
9、内部新闻更新。<br />
10、内部工具的’headers’;<br />
11、变更日志。<br />
12、团队要短小精悍。<br />
一个有趣的数字:平均每台Facebook的服务器8分钟就升级一次。<br />
一个有趣的事实:Facebook最忙的时候是万圣节后的那天。</p>
CU的perl大赛
2011-03-01T00:00:00+08:00
perl
http://chenlinux.com/2011/03/01/perl-game-of-chinaunix
<p>原帖地址:http://bbs.chinaunix.net/thread-1860259-1-1.html<br />
刚看到帖子,试着自己做做,首先必须承认做的过程是重新去翻过资料了……</p>
<ol>
<li>
<p><code class="highlighter-rouge">perl#!/usr/bin/perl myfunc {
# $x = ...
return $x?1:undef;
}</code></p>
</li>
<li>
<p>$x=()就是给$x赋了个undef;列表在标量上下文中取列表的最后一个元素;()是创建一个空列表;至于为啥取最后一个的原因,我猜测是采用的pop操作,所以从列表最后取值吧。</p>
</li>
<li>
<p><code class="highlighter-rouge">perl@x=(1,2,3,5,6,7,8);
@y=@z=();
for (0..$#x) {
push @y, $x[$_] if $x[$_+1]-1 != $x[$_];
push @z, $x[$_] if $x[$_-1]+1 != $x[$_];
}
for (0..$#z) {
print $z[$_]."-".$y[$_];
print "," unless $_ == $#z;
}</code></p>
</li>
<li>
<p>基本没用过@x[1]这个写法啊,猜测与$x[1]的区别会是在上下文上吧?一个标量一个列表。反正就这个题目的例子,print结果都是7</p>
</li>
<li>
<p>汗,不知道print <code class="highlighter-rouge">ls -lta</code>;算不算最短perl代码?</p>
</li>
<li>
<p><code class="highlighter-rouge">perl@x=(..)
for (@x) {
$sum += $_;
}
$avg = $sum / @x;
for (@x) {
print "$_ " if $_ > $avg;
}</code></p>
</li>
<li>
<p><code class="highlighter-rouge">perl$x = int(1 + rand 100);
while (<>) {
chomp;
exit unless /\d+/;
print "Too low\n" and next if $_ < $x;
print "Too high\n" and next if $_ > $x;
print "Right" and exit if $_ == $x;
}</code></p>
</li>
<li>
<p>不知道啥是无阻塞IO……</p>
</li>
</ol>
RT故障处理操作一例
2011-02-24T00:00:00+08:00
database
MySQL
perl
http://chenlinux.com/2011/02/24/delete-rt-attachment-from-mysql
<p>公司RT系统某工单页面无法打开。通过httpwatch发现是图片附件比较大,卡住了页面加载最终导致。<br />
询问当事人后,决定把图片删除掉。<br />
右键菜单查看图片url,是http://rt.domain.com/Ticket/Attachment/123456/654321/12.jpg这样的格式~<br />
于是在服务器的DocumentRoot下查找相关路径,发现Ticket/Attachment下只有一个文件dhandler,这是一段perl程序。<br />
相关部分如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">my</span> <span class="nv">$arg</span> <span class="o">=</span> <span class="nv">$m</span><span class="o">-></span><span class="nv">dhandler_arg</span><span class="p">;</span> <span class="c1"># get rest of path</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$arg</span> <span class="o">=~</span> <span class="s">'^(\d+)/(\d+)'</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$trans</span> <span class="o">=</span> <span class="nv">$1</span><span class="p">;</span>
<span class="nv">$attach</span> <span class="o">=</span> <span class="nv">$2</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">my</span> <span class="nv">$AttachmentObj</span> <span class="o">=</span> <span class="k">new</span> <span class="nn">RT::</span><span class="nv">Attachment</span><span class="p">(</span><span class="nv">$session</span><span class="p">{</span><span class="s">'CurrentUser'</span><span class="p">});</span>
<span class="nv">$AttachmentObj</span><span class="o">-></span><span class="nv">Load</span><span class="p">(</span><span class="nv">$attach</span><span class="p">)</span> <span class="o">||</span> <span class="nv">Abort</span><span class="p">(</span><span class="s">"Attachment '$attach' could not be loaded"</span><span class="p">);</span>
<span class="k">unless</span> <span class="p">(</span><span class="nv">$AttachmentObj</span><span class="o">-></span><span class="nv">id</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">Abort</span><span class="p">(</span><span class="s">"Bad attachment id. Couldn't find attachment '$attach'\n"</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">unless</span> <span class="p">(</span><span class="nv">$AttachmentObj</span><span class="o">-></span><span class="nv">TransactionId</span><span class="p">()</span> <span class="o">==</span> <span class="nv">$trans</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">Abort</span><span class="p">(</span><span class="s">"Bad transaction number for attachment. $trans should be"</span><span class="o">.</span><span class="nv">$AttachmentObj</span><span class="o">-></span><span class="nv">TransactionId</span><span class="p">()</span> <span class="o">.</span><span class="s">"\n"</span><span class="p">);</span>
<span class="p">}</span>
</code></pre>
</div>
<p>显然,该图片url对应的就是$trans=123456;$attach=654321。</p>
<p>再看RT/Attachment.pm,里面记录的是Attachments表的情况;再看其中的Create{}中相关部分如下:<br />
<code class="highlighter-rouge">perl
my $id = $self->SUPER::Create(
TransactionId => $args{'TransactionId'},
ContentType => $Attachment->mime_type,
ContentEncoding => $ContentEncoding,
Parent => $args{'Parent'},
Headers => $Attachment->head->as_string,
Subject => $Subject,
Content => $Body,
Filename => $Filename,
MessageId => $MessageId,
</code></p>
<p>基本可以通过工单的id、url里的transactionid和Filename来唯一确定这个特大的图片了。</p>
<p>(实际这个Filename已经在transactionid生成的时候可以无视掉了,参见RT/Transaction_Overlay.pm里的Create{}。所以url里不管最后一段写什么*.jpg,结果都一样)</p>
<div class="highlighter-rouge"><pre class="highlight"><code># mysql -uroot -p
> select * from rt3.Attachments where id='1234' and TransactionId='123456' and Filename='12.jpg';
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code> 然后屏幕开始哗哗的刷,全是-------
</code></pre>
</div>
<p>因为把图片内容存在Content字段里,显示就全是-了。</p>
<p>不过还是担心,万一这个-不是想象中的呢?</p>
<p>于是去找binlog。通过 <code class="highlighter-rouge">strings binlog.* | awk '/12.jpg/ && /'$Date'/{print NR}'</code> 找到当初create的记录(好在不是啥繁忙的系统,不然这种方法能被DBA鄙视死……),然后通过 <code class="highlighter-rouge">awk 'NR>a && NR<b'</code> 的方式查看create记录附件的其他内容。果然在Content字段,有几百行乱码,最开头就是JFIF,也就是jpg的图片格式。</p>
<p>最后 <code class="highlighter-rouge">update rt3.Attachments set Content='' where id='1234' and TransactionId='123456' and Filename='12.jpg';</code>删除这个超大图片,浏览页面就变快多啦~~</p>
awk一例
2011-02-24T00:00:00+08:00
bash
awk
http://chenlinux.com/2011/02/24/a-awk-example-about-gsub-function
<p>一个小需求,某目录下有五万多个模板文件,其中大概两万个是链接文件。当碰到如下情况:<br />
lrwxrwxrwx 1 nobody nobody 59 Dec 27 15:06 10000053.mod -> /var/www/html/category/model/10000050.mod<br />
就需要创建同名的另一个模板文件10000053.wap文件,指向10000050.wap。<br />
怎么做?<br />
我用了如下命令:<br />
<code class="highlighter-rouge">bash
ls -l /var/www/html/category/model | awk '$1~/^l/ && $9~ /mod$/ {gsub(/mod$/,"wap",$9);gsub(/mod$/,"wap",$NF);system("rm -f "$9" && ln -s "$NF" "$9)}'</code><br />
不过从效果来看,使用gsub函数后速度慢了不少,这5万个文件花了几分钟。</p>
<hr />
<p>另,在CU上看到另一个文件操作的shell考题,据说是腾讯的题。修改某目录下(含子目录)所有.shell文件为.sh。我的思路和上头的类似。不过在微博上看到一个超级不错的写法,记录一下:<br />
<code class="highlighter-rouge">bashrename .shell .sh `find ./ -name *.shell</code>`</p>
没事试试RH的技能测试
2011-02-22T00:00:00+08:00
linux
http://chenlinux.com/2011/02/22/redhat-tech-questions
<p>上redhat官网看资料的时候想起来上面有RHCE报名前的技能水平测试(用来预估水平,省掉一些基础课程的)。然后做了一下试试。地址如右:<a href="http://www.redhat.com/explore/pre-assessment">http://www.redhat.com/explore/pre-assessment</a><br />
结果如下:</p>
<table border="0" cellspacing="0">
<thead>
<tr>
<th>Topic</th>
<th>Evaluation</th>
</tr>
</thead>
<tbody>
<tr>
<td>Essential Command-Line Operations</td>
<td>Deep Understanding</td>
</tr>
<tr>
<td>Managing Simple Partitions and Filesystems</td>
<td>Some Understanding</td>
</tr>
<tr>
<td>Managing User Accounts</td>
<td>Substantial Knowledge</td>
</tr>
<tr>
<td>Tuning and Maintaining the Kernel</td>
<td>Some Understanding</td>
</tr>
<tr>
<td>Enhance User Security</td>
<td>Familiarity</td>
</tr>
<tr>
<td>BASH Scripting and Tools</td>
<td>Familiarity</td>
</tr>
</tbody>
</table>
<p><em>* The results represent a subset of the knowledge in the curriculum.</em></p>
<p><em> Recommendation
RHCSA Rapid Track Course with Exam (RH200)
Red Hat System Administration III with RHCSA and RHCE Exams (RH255)</em><br />
发现用win做个人桌面使用,对考rhc*还是有影响的——不少题是窗口应用的~</p>
浏览器连接数的小区别
2011-02-19T00:00:00+08:00
CDN
http://chenlinux.com/2011/02/19/conns-diff-between-browsers
<p>读百度UEO博客的文章《<a href="http://www.baiduux.com/blog/2011/02/15/browser-loading/" target="_blank">浏览器的加载与页面性能优化</a>》,其中关于浏览器对单个域名连接数有一段描述,与一般的概述稍有差别。由此可见像百度这种级别的公司,对性能细节抓到什么程度——让我想起之前在腾讯大讲堂里看到的”页面代码大小要求是MTU倍数”。</p>
<p>这段文字如下:<br />
在HTTP/1.1协议下,单个域名的最大连接数在IE6中是2个,而在其它浏览器中一般4-8个,而整体最大链接数在30左右</p>
<p>而在HTTP/1.0协议下,IE6、7单个域名的最大链接数可以达到4个,在Even Faster Web Sites一书中的11章还推荐了对静态文件服务使用HTTP/1.0协议来提高IE6、7浏览器的速度</p>
apache的rewrite伪静态化问题一例
2011-02-16T00:00:00+08:00
apache
http://chenlinux.com/2011/02/16/pseudo-static-problem-of-js-rewrite
<p>某应用系统有一个产品翻页浏览,为了利于搜索引擎,准备把/search.html?param=1-2-3-4-5做伪静态化,变成/search/1-2-3-4-5.html的url显示。</p>
<p>想来很简单,确认apache有mod_rewrite后,加入如下配置,重启即可:<br />
```apache</p>
<ifmodule mod_rewrite.c="">
RewriteEngine on
RewriteCond %{HTTP_HOST} ^example\.domain\.com [NC]
RewriteCond %{REQUEST_URI} ^/search/.*\.html
RewriteRule ^/search/(.*)\.html /search.html?param=$1 [P,L]
</ifmodule>
<p>```<br />
不过很不幸的事情出现了:不管点击页面搜索结果的哪个页码,看到的永远都是第一页的内容!!</p>
<p>是搜索程序有问题么?试着直接访问/search.html?param=1-2-3-4-6,能看到之后某页的新内容。</p>
<p>于是打开apache的rewritelog看看究竟:<br />
<code class="highlighter-rouge">apacheRewriteLog /www/admin.game.china.com/logs/rewrite.log
RewriteLogLevel 9</code><br />
在rewrite.log中看到如下记录:<br />
<code class="highlighter-rouge">apache[rid#13c4e40/initial] (2) init rewrite engine with requested uri /search/0-0-0-0-0-0-0-0-0-2.html
[rid#13c4e40/initial] (3) applying pattern '^/search/(.*)\.html' to uri '/search/0-0-0-0-0-0-0-0-0-2.html'
[rid#13c4e40/initial] (4) RewriteCond: input='example.domain.com' pattern='^example\.domain\.com' => matched
[rid#13c4e40/initial] (4) RewriteCond: input='/search/0-0-0-0-0-0-0-0-0-2.html' pattern='^/search/.*\.html' => matched
[rid#13c4e40/initial] (2) rewrite /search/0-0-0-0-0-0-0-0-0-2.html -> /search.html?param|0-0-0-0-0-0-0-0-0-2
[rid#13c4e40/initial] (3) split uri=/search.html?param|0-0-0-0-0-0-0-0-0-2 -> uri=/search.html, args=param|0-0-0-0-0-0-0-0-0-2
[rid#13c4e40/initial] (2) local path result: /search.html
[rid#13c4e40/initial] (2) prefixed with document_root to /www/example.domain.com/search.html
[rid#13c4e40/initial] (1) go-ahead with /www/example.domain.com/search.html [OK]</code><br />
看起来似乎没有问题……</p>
<p>为了更细致一点,在测试环境中用单进程方式启动apache,通过strace来跟踪httpd。更明确看到了rewrite之后的内部请求”GET /search.html?param=0-0-0…“完全没有问题。</p>
<p>于是找开发同事商量,询问这个.html?param=是如何完成翻页功能的。结果得知是html中有javascript,翻页就是通过js来完成的。</p>
<p>恍然大悟!js是在浏览器端完成解析工作的,那么在apache里rewrite的时候传输过去的args没有起到作用,服务器返回的永远是默认的html内容,即第一页内容。同事修改程序,将翻页方式改成另外的jsp来完成,我再修改rewrite规则到jsp上。一切OK了!</p>
linux内核编译升级
2011-02-14T00:00:00+08:00
linux
http://chenlinux.com/2011/02/14/compile-and-upgrading-of-the-linux-kernel
<p>N年没更新的ipvsadm终于在今年春节前更新了,正好手头有lvs的任务,赶紧试试。lvs上说的很清楚,ipvsadm的1.2.26版仅工作于linux kernel2.6.28以上版本。所以首先要把现有的2.6.18的linux kernel升级。</p>
<ol>
<li>
<p>从kernel.org上获取高版本的kernel原文件:<br />
wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.37.tar.bz2<br />
tar jxvf linux-2.6.37.tar.bz2 -C /usr/src</p>
</li>
<li>
<p>做好源代码连接<br />
ln -s /usr/src/linux-2.6.37 /usr/src/linux</p>
</li>
<li>
<p>开始选择编译参数<br />
cd /usr/src/linux<br />
make mrproper #这一步是清除可能存在的其他内核编译结果<br />
make menuconfig #采用字符界面选择,初次操作的就别改什么了。<br />
make bzImage #生成vmlinuz<br />
make modules<br />
make modules_install #生成模块<br />
make install #把生成的System.map/initrd/vmlinuz等都mv到/boot下,并修改grub配置</p>
</li>
<li>
<p>重启<br />
sed -i ‘s/default=1/default=0/’ /boot/grub/menu.lst<br />
reboot</p>
</li>
</ol>
<p>嗯,很好,然后等待,十分钟过去,依然ping不通(这么简单就搞定,我也懒得写这篇博文啦)……赶紧接显示器看看进展。启动界面停留在如下画面:<br />
Unable to access resume device (LABEL=SWAP-sda9)<br />
mount : could not find filesystem ‘/dev/root’<br />
setup other filesystem<br />
setting up now root fs<br />
set up root :moving /dev faild:No such file or directory<br />
no fstab.sys,mounting inernal defaults<br />
setuproot:error mounting /proc :No such file or directory<br />
setuproot:error mounting /sys:No such file or directory<br />
switching to new root and running init<br />
umounting old /dev<br />
umounting old /proc<br />
umounting old /sys<br />
switchroot : mount faild : No such file or directory<br />
kernel panic:not syncing :attempted to kill init<br />
call trace<br />
sysfs系统无法挂载……原来linux kernel2.6.3*中,在menuconfig中有个很重要的选项:<br />
enable deprecated sysfs features which may confuse old userspace tools<br />
help文档对这个选项的解释是:“<strong>Do not say Y, if the original kernel, that came with your distribution, has this option set to N.</strong>”</p>
<p><span style="color: #ff0000;">很不幸,RHEL5的/usr/src/kernels/2.6.18-92.el5-x86_64/.config中压根就没有CONFIG_SYSFS_DEPRECATED这行……所以必须选上这个选项。</span></p>
<p>选择老内核进入系统,重新来过一次编译,除了这个选项以外一切相同。重启就成功进入了!</p>
直接操作xen虚拟机镜像的办法
2011-01-26T00:00:00+08:00
cloud
xen
http://chenlinux.com/2011/01/26/direct-operate-xen-vm-image
<p>一台xen虚拟机,root密码忘记了,必须进入single模式修改root密码。步骤很熟练,xm shutdown domain && xm create -c domain——但是出问题了——没出现grub的启动界面,直接进入系统启动过程了!</p>
<p>无法从正常操作流程上搞定,那么换一个思路,直接操作镜像文件(谢天谢地,还好是虚拟机),采用loop方式挂载镜像,然后上去改文件好了~</p>
<p>方法如下:</p>
<p>mount -o loop,offset=32256 /xen/disk.img /mnt<br />
然后看/mnt/下的grub.conf,timeout=0,修改成timeout=5,保存退出。umount /mnt后重新使用xm cre -c就可以看到grub界面了~~</p>
<p>现在解释一下这个offset=32256是怎么来的。因为如果不加这串会报出“mount: you must specify the filesystem type”错误。</p>
<p>1、先file确定img文件,如下:<br />
<code class="highlighter-rouge">bash[root@localhost xen]# file /xen/cvs_backup
/xen/cvs_backup: x86 boot sector;
partition 1: ID=0x83, active, starthead 1, startsector 63, 208782 sectors;
partition 2: ID=0x8e, starthead 0, startsector 208845, 62701695 sectors, code offset 0x48</code><br />
可以看到这个镜像文件其实被格式化成了两个分区,其中第一个分区的起始块位置是63;<br />
2、用fdisk确定具体的units,如下:<br />
<code class="highlighter-rouge">bash[root@localhost xen]# fdisk -lu /xen/cvs_backup
last_lba(): I don't know how to handle files with mode 81ed
You must set cylinders.
You can do this from the extra functions menu.
Disk /xen/cvs_backup: 0 MB, 0 bytes
255 heads, 63 sectors/track, 0 cylinders, total 0 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/xen/cvs_backup1 * 63 208844 104391 83 Linux
/xen/cvs_backup2 208845 62910539 31350847+ 8e Linux LVM
Partition 2 has different physical/logical endings:
phys=(1023, 254, 63) logical=(3915, 254, 63)</code><br />
可见每个块的大小是512字节。那么虚拟机镜像文件对应的真实起始字节位置就是63*512=32256了。</p>
图片分离小故障一例
2011-01-20T00:00:00+08:00
linux
http://chenlinux.com/2011/01/20/problem-about-move-crossing-partition
<p>有需求对某应用的图片和页面内容进行拆分。计划进行的很顺利,存储设备上成功分成了/vol/html和/vol/image,然后分别挂载在应用服务器上发布为html.domain.com和image.domain.com。但在恢复应用运行后出现一个问题:新图片的上传总是提示失败。</p>
<p>查看服务器上的日志,还好jvm-default.log记录的相当详细。上传程序首先在html.domain.com/dynamic/domain<em>/下创建一个tmp/,上传的图片</em>.jpg和缩略图s_<em>.jpg就暂存在该tmp/下。然后再mv到相应的image.domain.com/domain</em>/$date下。图片上传到tmp是成功的,特意选择一个比较偏僻的domain9,由系统来新建这个image.domain.com/domain*/$date,也成功了。</p>
<table>
<tbody>
<tr>
<td>开发同事去检查程序,试图提供更详细的报错信息,然后发现报错的地方写的是if (!TmpPicReNameNewPic</td>
<td> </td>
<td>!TmpSmallPicReNameNewSmallPic ) {*};</td>
</tr>
</tbody>
</table>
<p>于是想到有一种可能,虽然存储还是同一个存储,但是分成了两次mount,或许linux系统就将该存储当成了两个磁盘,而rename方法在linux的实现是不可以跨磁盘操作的。见man文档:<br />
oldpath and newpath are not on the same mounted filesystem.<br />
(Linux permits a filesystem to be mounted at multiple points, but rename(2)<br />
does not work across different mount points, even if the same filesystem is<br />
mounted on both.)<br />
解决办法有两种,一种是修改程序,把rename改成真正的mv(即先cp,然后rm) ;另一种是想办法只挂载一次存储。考虑到这个猜测有可能不正确,折腾程序比较麻烦,正好这次图文拆分也不是彻底的分离到不同服务器上,而是存储上的两个路径而已。决定先mount NetAppIp:/vol/ /mnt;ln -s /mnt/html /www/html.domain.com;ln -s /mnt/image /www/image.domain.com;重启服务再测试上传,果然成功了!</p>
awk的效率
2011-01-20T00:00:00+08:00
bash
awk
http://chenlinux.com/2011/01/20/awk-efficiency
<p>偶然和某人谈到日志处理。最简单常见的需求,日志中访问量最大的前十个IP及其访问次数。</p>
<table>
<tbody>
<tr>
<td>最常见的shell命令:cat access.log</td>
<td> cut -d ‘ ‘ -f 4 </td>
<td>sort</td>
<td>uniq -c</td>
<td>sort -nr</td>
<td>head</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>我最常用的awk命令:awk ‘{a[$4]++}END{for(i in a){print a[i],i}}’ access.log</td>
<td>sort -nr</td>
<td>head</td>
</tr>
</tbody>
</table>
<p>对方表示上一种速度最快,而我说是下一种。</p>
<p>最后找到一个13G大小的access.log,用time命令分别检测命令用时。结果处理一个13GB的日志,shell花了13分钟,awk花了1分半钟……</p>
读《基于动态内容的缓存加速技术》笔记
2011-01-13T00:00:00+08:00
CDN
http://chenlinux.com/2011/01/13/learning-accel-dynamic-data
<p>《程序员》2010年11月刊的86-89四页,刊登了F5售前技术顾问,原VIACDN(TOM的CDN部门,后来独立运营)架构师徐超的文章《基于动态内容的缓存加速技术——F5 Web Accelerator产品技术剖析》。</p>
<p>文章的前三分之一内容,主要是讲述RFC2616和一般浏览器、服务器的具体实现。已经比较熟悉了,略过……</p>
<p>然后进入关键内容。</p>
<p>“让浏览器缓存能够为动态页面工作”:<br />
A. 修改Transfer-Encoding: chunked的内容输出Content-Length。<br />
这个squid已经做到了。其实质就是在获取url的body时,累加每个chunk的size,直到碰上MSB为0的last-chunk标记。<br />
B. 添加Last-Modified。<br />
这个squid差一步。squid实质已经获取了文件在本地的mtime,但只在源header不存在last-modified的时候才用于LM-factor算法。<br />
C. 删除Expires。<br />
大概是为了统一成HTTP/1.1的header,所以选择了删除expires?或者就是纯粹了节省代码吧……<br />
D. 修改Cache-Control。有Expires按这个设定max-age,或者按配置设定;有private的修改成public;添加must-revalidated。<br />
这个squid用header_access和header_replace就能完成。</p>
<p>“Web Accelerator如何识别动态页面的更新并保持更新”:<br />
A. 预加载。<br />
这个通过squid的补丁html prefetching可以完成<br />
B. 根据上一章节的修改结果,浏览器对”动态html”每次都可以发送IMS到F5。F5向源站请求该html,并比对缓存内容。<br />
其他过程和一般的IMS过程一样,关键在比对缓存内容。如果这个”动态”只是html文字内容的更新,还是正常范围;如果”动态交互”的结果还包括其他CSS/JS/JPG/GIF的变动,这个功能就比较有用了。<br />
根据预加载的功能,每次确认html时重新预比对。考虑到CSS/JS/JPG/GIF一般来说都是会输出Last-Modified的,这个比对返回结果应该比较快。都没问题的话,跳过这步;<br />
如果有部分文件mtime也变动了,开始预加载,并另存为一个带有版本号的url(比如logo.gif;pv=***),用’,’而不能用’?’,因为?在浏览器上是不缓存的;<br />
同时修改主文件html的内容,替换url链接为新版本号的url。最后发送这个新版本html给浏览器。<br />
这种版本号控制的方法也节省了缓存更新时的删除操作IO,而统一交给LRU之类的完成。<br />
想到LRU,一个疑问:当缓存不足时,假如logo.gif已经被prune出去了,那这个时候的F5怎么办?几个猜测办法:<br />
1、浪费一点带宽和时间,以新版本号url的形式重新进入缓存;<br />
2、用一个版本号/mtime的K-V数据库完成url版本控制;<br />
3、F5作为特定类型给某些网站使用,就不考虑海量文件的问题。<br />
最后,我还是觉得url版本控制这种事情,还是由网站开发人员来做CMS比较靠谱。</p>
<p>“反向动态代理”<br />
上面的方式,确实保证了数据”实时”性,而且充分利用了浏览器缓存,但一次刷新引发F5和源站之间几十上百次的比对,还是比较郁闷的。所以当网站的”动态”结构比较清晰时,比如一个论坛,明确知道帖子列表变动就是因为有人发帖了;而且可以从发帖的POST请求url里推导出版面url的,可以采用这种技术。<br />
一旦接收到POST请求,F5自动根据配置purge版面url的缓存。而不用去比对。</p>
<p>“MultiConnect技术”<br />
因为浏览器对同一域名并发连接很少,所以F5可以自动替换html里的url,根据配置把image.x.com换成img1.x.com/img2.x.com/img3.x.com……前提是你的DNS上确实有这些解析。<br />
不过要注意,域名太多也会拖慢浏览器速度的。dns解析需要时间。</p>
<p>总之,个人感觉这些技术还是比较现实的,但都用很强的应用场景针对性。呵呵~</p>
resin-status
2011-01-12T00:00:00+08:00
monitor
resin
http://chenlinux.com/2011/01/12/resin-status
<p>和apache、nginx一样,resin也自带了一个比较简易的status模块,只需要在resin.conf里配置就行了。在里添加如下一段:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nt"><servlet-mapping</span> <span class="na">servlet-class=</span><span class="s">'com.caucho.servlets.ResinStatusServlet'</span><span class="nt">></span>
<span class="nt"><url-pattern></span>/resin-status<span class="nt"></url-pattern></span>
<span class="nt"><init</span> <span class="na">enable=</span><span class="s">"read"</span><span class="nt">/></span>
<span class="nt"></servlet-mapping></span>
</code></pre>
</div>
<p>重启resin即可。</p>
<p>然后curl访问http://127.0.0.1:8080/resin-status就可以看到输出了。</p>
<p>用浏览器的话,大致如下图:<br />
<img src="/images/uploads/resin.jpg" alt="resin" /></p>
<p>可以关注线程数、连接数、内存使用大小和请求处理数。</p>
<p>如果要做监控的话,直接从html里获取相应数值即可。</p>
<p>唯一需要稍微注意的是请求处理数。因为其他的都是AVERAGE,只有这个是COUNTER型的。要是想很方便的看到rps。可以采用如下方法查看:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># a=`curl -s http://127.0.0.1:8081/resin-status|awk -F/ '/Invocation/{print $NF}'`;sleep 1;curl -s http://10.168.168.56:8081/resin-status|awk -F/ '/Invocation/{print($NF-'$a'}'</span>
718
</code></pre>
</div>
<p>这里比较有趣的是因为这行的数据本身是有个括号的,所以就有了各种各样的写法和报错了。一并贴上来,可以体会一下awk的系统变量/内部函数/字符类型的用法:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># a=`curl -s http://127.0.0.1:8080/resin-status|awk -F/ '/Invocation/{print $NF}'`;sleep 10;curl -s http://127.0.0.1:8080/resin-status|awk -F/ '/Invocation/{print $NF-'$a'}'</span>
awk: cmd. line:1: /Invocation/<span class="o">{</span>print <span class="nv">$NF</span>-3949612<span class="o">)}</span>
awk: cmd. line:1: ^ syntax error
<span class="c"># a=`curl -s http://127.0.0.1:8080/resin-status|awk -F/ '/Invocation/{print $NF}'`;sleep 10;curl -s http://127.0.0.1:8080/resin-status|awk -F/ '/Invocation/{print $NF-'"$a"'}'</span>
awk: cmd. line:1: /Invocation/<span class="o">{</span>print <span class="nv">$NF</span>-4025373<span class="o">)}</span>
awk: cmd. line:1: ^ syntax error
<span class="c"># a=`curl -s http://127.0.0.1:8080/resin-status|awk -F/ '/Invocation/{print $NF}'`;sleep 10;curl -s http://127.0.0.1:8080/resin-status|awk -F/ '/Invocation/{print $NF-"'$a'"}'</span>
7607
<span class="c"># a=`curl -s http://127.0.0.1:8080/resin-status|awk -F/ '/Invocation/{print substr($NF,0,length($NF)-1)}'`;sleep 10;curl -s http://127.0.0.1:8080/resin-status|awk -F/ '/Invocation/{print $NF-'$a'}'</span>
7927
<span class="c"># a=`curl -s http://127.0.0.1:8080/resin-status|awk -F/ '/Invocation/{print $NF}'`;sleep 10;curl -s http://127.0.0.1:8080/resin-status|awk -F/ '/Invocation/{print($NF-'$a'}'</span>
7212
</code></pre>
</div>
HTTP的auth请求模拟
2011-01-11T00:00:00+08:00
web
http://chenlinux.com/2011/01/11/construct-http-auth-header
<p>开发同事需要在程序中调用一个“安全级别比较高”的url,起初没觉得有啥问题,我们wget或者curl的时候,按照标准url的格式(即’请求方法://用户名:密码@域名/文件路径’)写就完全OK了。不过很快同事转来了报错:</p>
<p><span style="color:#000000;font-family:Verdana;font-size:x-small;"><span style="font-family:Verdana;font-size:x-small;"><span style="font-family:Verdana;font-size:x-small;">java.io.IOException: Server returned HTTP response code: 401 for URL: <a href="http://gundong:gdxw6354@editornew.china.com/interface/addcategory.php?parentid=2&id=2&name=gbox_%D0%C7%BC%CA%D5%F9%B0%D4&groupname=game&code=satrcraft&m=09286f9d135d5debe7052bea42a27eef">http://test:test1234@test.domain.com/interface/addcategory.php?parentid=2&id=2&name=gbox_%D0%C7%BC%CA%D5%F9%B0%D4&groupname=game&code=satrcraft&m=09286f9d135d5debe7052bea42a27eef</a></span></span></span><br />
原来用的是IO的方式,我用telnet模拟一下,结果还真是这样:<br />
```bash[root@cms ~]# telnet test.domain.com 80<br />
Trying 123.124.125.126…<br />
Connected to test.domain.com (123.124.125.126).<br />
Escape character is ‘^]’.<br />
GET http://test:test1234@test.domain.com/interface/addcategory.php?parentid=2&id=2&name=gbox_%D0%C7%BC%CA%D5%F9%B0%D4&groupname=game&code=satrcraft&m=09286f9d135d5debe7052bea42a27eef HTTP/1.0</p>
<p>HTTP/1.1 401 Authorization Required<br />
Date: Tue, 11 Jan 2011 03:39:37 GMT<br />
Server: Apache/1.3.37 (Unix) PHP/4.4.9<br />
WWW-Authenticate: Basic realm=”CMS-Testdotcom”<br />
Connection: close<br />
Content-Type: text/html; charset=iso-8859-1```<br />
查了一下HTTP协议,原来auth是走的另外一个header完成Authorization,其格式是Authorization: Basic ‘encoded_base64(user:passwd)’。服务器会自动的用decoded_base64()解析字符串得到真正的用户名和密码。原来wget和curl这些工具不单单是发个请求这么简单啊~~</p>
<p>重新试验,先计算test:test1234的base64值:<br />
```bash[root@cms ~]# echo test:test1234|openssl base64<br />
dGVzdDp0ZXN0MTIzNAo=<br />
[root@cms ~]# telnet test.domain.com 80<br />
Trying 123.124.125.126…<br />
Connected to test.domain.com (123.124.125.126).<br />
Escape character is ‘^]’.<br />
GET http://test.domain.com/interface/addcategory.php?parentid=2&id=2&name=gbox_%D0%C7%BC%CA%D5%F9%B0%D4&groupname=game&code=satrcraft&m=09286f9d135d5debe7052bea42a27eef HTTP/1.0<br />
Authorization: Basic dGVzdDp0ZXN0MTIzNAo=</p>
<p>HTTP/1.1 200 OK<br />
Date: Tue, 11 Jan 2011 05:21:32 GMT<br />
Server: Apache/1.3.37 (Unix) PHP/4.4.9<br />
X-Powered-By: PHP/4.4.9<br />
Connection: close<br />
Content-Type: text/html<br />
2 || 2|| gbox_星际争霸 || satrcraft || game <br />09286f9d135d5debe7052bea42a27eef<br />2Connection closed by foreign host.```<br />
果然就可以了~~</p>
mysqlreport指南
2011-01-10T00:00:00+08:00
database
MySQL
http://chenlinux.com/2011/01/10/mysqlreport-guide
<p>mysqlreport是mysql性能监测时最常用的工具,对了解mysql运行状态和配置调整都有很大的帮助。找了一些mysql的资料,发现大多数是关于php+mysql开发的,服务配置基本就是固定的几条。干脆找上mysqlreport的官网,啃下来这篇指南。翻译都是随着我个人的语言习惯,对直接能用mysql命令上看到结果的英文则保留下来。方便以后查找:</p>
<p>原文地址:<a href="http://hackmysql.com/mysqlreportguide">http://hackmysql.com/mysqlreportguide</a></p>
<h1 id="mysqlreport">《mysqlreport指南》</h1>
<p>本指南的写作目的是解释mysqlreport上出具的各种数据。通过指南,你可以在阅读完mysqlreport的报告后,完整的解释并且理解一个最根本的问题——而这也正是你使用mysqlreport的目的所在——(我的)MySQL服务器到底运行的怎么样?</p>
<p>当前的mysqlreport版本会自动生成一份尽可能完整的数据报表,最多可达14节121行。您不必再像以前那样输入各种各样的参数定义了。不过如果您的mysql服务配置上关闭了某些部分的话,报表的相关部分也会随之关闭。比如,你关闭了<code class="highlighter-rouge">query cache</code>,那么mysqlreport的第4节’Query Cache’也就不存在了。所以报表长度是浮动的。</p>
<p>为了便于理解和观察,本指南采用了对一份完整报表做出逐行解释的方法。下面就是将要被详细解析的那份报表,为了方便,在最前端加上了行号。</p>
<pre><code class="language-mysql">1 MySQL 5.0.3 uptime 0 0:34:26 Fri Sep 1 19:46:02 2006
2
3 __ Key _________________________________________________________________
4 Buffer used 380.00k of 512.00M %Used: 0.07
5 Current 59.32M %Usage: 11.59
6 Write hit 97.04%
7 Read hit 99.58%
8
9 __ Questions ___________________________________________________________
10 Total 98.06k 47.46/s
11 DMS 81.23k 39.32/s %Total: 82.84
12 QC Hits 16.58k 8.02/s 16.91
13 COM_QUIT 200 0.10/s 0.20
14 Com_ 131 0.06/s 0.13
15 -Unknown 82 0.04/s 0.08
16 Slow 5 s 0 0.00/s 0.00 %DMS: 0.00 Log: ON
17 DMS 81.23k 39.32/s 82.84
18 SELECT 64.44k 31.19/s 65.72 79.33
19 INSERT 16.75k 8.11/s 17.08 20.61
20 UPDATE 41 0.02/s 0.04 0.05
21 REPLACE 0 0.00/s 0.00 0.00
22 DELETE 0 0.00/s 0.00 0.00
23 Com_ 131 0.06/s 0.13
24 change_db 119 0.06/s 0.12
25 show_fields 9 0.00/s 0.01
26 show_status 2 0.00/s 0.00
27
28 __ SELECT and Sort _____________________________________________________
29 Scan 38 0.02/s %SELECT: 0.06
30 Range 14 0.01/s 0.02
31 Full join 3 0.00/s 0.00
32 Range check 0 0.00/s 0.00
33 Full rng join 0 0.00/s 0.00
34 Sort scan 14 0.01/s
35 Sort range 26 0.01/s
36 Sort mrg pass 0 0.00/s
37
38 __ Query Cache _________________________________________________________
39 Memory usage 17.81M of 32.00M %Used: 55.66
40 Block Fragmnt 13.05%
41 Hits 16.58k 8.02/s
42 Inserts 48.50k 23.48/s
43 Prunes 33.46k 16.20/s
44 Insrt:Prune 1.45:1 7.28/s
45 Hit:Insert 0.34:1
46
47 __ Table Locks _________________________________________________________
48 Waited 1.01k 0.49/s %Total: 1.24
49 Immediate 80.04k 38.74/s
50
51 __ Tables ______________________________________________________________
52 Open 107 of 1024 %Cache: 10.45
53 Opened 118 0.06/s
54
55 __ Connections _________________________________________________________
56 Max used 77 of 600 %Max: 12.83
57 Total 202 0.10/s
58
59 __ Created Temp ________________________________________________________
60 Disk table 10 0.00/s
61 Table 26 0.01/s Size: 4.00M
62 File 3 0.00/s
63
64 __ Threads _____________________________________________________________
65 Running 55 of 77
66 Cache 0 %Hit: 0.5
67 Created 201 0.10/s
68 Slow 0 0.00/s
69
70 __ Aborted _____________________________________________________________
71 Clients 0 0.00/s
72 Connects 8 0.00/s
73
74 __ Bytes _______________________________________________________________
75 Sent 38.46M 18.62k/s
76 Received 7.98M 3.86k/s
77
78 __ InnoDB Buffer Pool __________________________________________________
79 Usage 3.95M of 7.00M %Used: 56.47
80 Read hit 99.99%
81 Pages
82 Free 195 %Total: 43.53
83 Data 249 55.58 %Drty: 0.00
84 Misc 4 0.89
85 Latched 0 0.00
86 Reads 574.56k 0.6/s
87 From file 176 0.0/s 0.03
88 Ahead Rnd 4 0.0/s
89 Ahead Sql 2 0.0/s
90 Writes 160.82k 0.2/s
91 Flushes 1.04k 0.0/s
92 Wait Free 0 0/s
93
94 __ InnoDB Lock _________________________________________________________
95 Waits 0 0/s
96 Current 0
97 Time acquiring
98 Total 0 ms
99 Average 0 ms
100 Max 0 ms
101
102 __ InnoDB Data, Pages, Rows ____________________________________________
103 Data
104 Reads 225 0.0/s
105 Writes 799 0.0/s
106 fsync 541 0.0/s
107 Pending
108 Reads 0
109 Writes 0
110 fsync 0
111
112 Pages
113 Created 23 0.0/s
114 Read 226 0.0/s
115 Written 1.04k 0.0/s
116
117 Rows
118 Deleted 25.04k 0.0/s
119 Inserted 25.04k 0.0/s
120 Read 81.91k 0.1/s
121 Updated 0 0/s
</code></pre>
<h1 id="section">报表抬头:第1行</h1>
<p>本行包括三部分信息:MySQL版本,MySQL运行时间,服务器当前时间。显示版本是为了提醒有些功能可能这台mysql没有;显示运行时间是为了评估报表数据的精准度。事实上,如果一台mysql运行时间连几个小时都没有的,生成的很可能是一份扭曲并且充满误导意见的报表。甚至几个小时都不够,因为它可能是在半夜没啥访问量的时候运行的几个小时。所以建议最少要运行过一个24小时,时间越长越能反应真实情况。<br />
在本例中,运行时间只有34分钟,所以报表也不具有真实的代表意义。</p>
<h1 id="section-1">索引报表:3-7行</h1>
<p>索引报表是第一个主要章节,因为索引(key或者说index)对于mysql数据库,是最最最重要的。虽然报表不可能直接告诉你这个库的索引好还是不好,但它能告诉你这个索引缓冲区(key buffer)被利用的怎么样了。<br />
注意:本报表仅汇总默认的MyISAM表的共享key buffer信息,而不会管管理员自建的其他空间。</p>
<h1 id="section-2">缓冲区使用情况:第4行</h1>
<p>对于mysql,我们的第一个问题就是:到底用了多少key buffer?如果不太多,没问题~因为mysql只会在有需求的时候才分配系统内存给key buffer。也就是说,my.cnf中定义了<code class="highlighter-rouge">key_buffer_size=512M</code>,不代表mysql启动时就创建一个512M大小的key buffer。</p>
<p>本行显示的,是mysql曾经使用过的key buffer峰值大小。而事实上,mysql应该用的更少,或者诡异的更多。这个更多的情况,mysql的术语叫“高水位”这个情况和my.cnf里的<code class="highlighter-rouge">key_buffer_size</code>是否足够大密切相关。当“水位”已经达到80-90%的时候,赶紧加大你的<code class="highlighter-rouge">key_buffer_size</code>吧。</p>
<p>注意:永远不用“担心”这个值超过95%,mysql文档指出,key buffer中的一部分会被mysql主程序用于内部数据结构,这些是mysqlreport无法统计的内容。所以,所谓的95%,其实已经是100%了……</p>
<h1 id="section-3">当前情况:第5行</h1>
<p>这行只有在mysql版本高于4.1.2时出现,因为之前mysql的<code class="highlighter-rouge">show status</code>中没有<code class="highlighter-rouge">key_blocks_unused</code>。这行数据显示的是mysql当前使用的key buffer大小。如果上行的used%太大的话,那么这行必然不会超过used,除非碰上那个传说中的bug了。综合这两行,相信对<code class="highlighter-rouge">key_buffer_size</code>的设置是否合理就有谱了~</p>
<p>本例中,mysql使用了60M的key buffer(12%),这就很不错,离满负荷运行还早着呢。</p>
<h1 id="section-4">写命中:第6行</h1>
<p>从本质上说,索引是基于内存的。因为访问内存的速度比硬盘快太多了。不过,mysql从磁盘里进行一点点读写操作总是不可避免的。</p>
<p>这行数据显示了写索引的效率(具体意思是:写入磁盘的key与写入内存的key的比值)。这个值没有什么参考答案,而是取决于业务类型。如果mysql主要执行的是update/insert之类的操作,那么正常比值接近0%;如果执行的select居多,那比值超过90%也是正常的。不过如果你看到的是一个负数,那说明mysql总是在往那个慢的要死的磁盘里写索引,这就很不妙了。</p>
<p>要想知道到底比值正常与否,请参考之后的DMS报表内容。</p>
<h1 id="section-5">读命中:第7行</h1>
<p>比写命中重要多了的就是读命中。同样,这个值就是读自磁盘的key与读自内存的比值。这个比值最好别低于99%!!再低就有问题了——很可能就是key buffer太小。mysql没法从内存里读到,只好找硬盘了……<br />
当然,如果你刚重启过一次mysql,那在一两个小时内,命中率低一点也是正常的。</p>
<h1 id="section-6">请求报表:9-26行</h1>
<p>第二个主要章节。它展示了很多关于mysql在做什么以及做的怎么样的内容。请求(question)包括SQL查询(query),也包括mysql协议通讯。大家经常关注的一个性能是mysql的qps(每秒执行查询数)。不过从一个更广泛的眼光看来,这个衡量标准其实是很随意的……mysql需要处理很多其他的请求。本报表试图展现的,就是这么一个更完整的内容。</p>
<h1 id="section-7">总值:第10行</h1>
<p>本行第一列,回答自运行起mysql一共处理多少请求,第二列,得出自运行起平均每秒钟处理多少请求。大家可能以为第二列这个值就是我们想要的qps了。但mysql真的做了这么多事情么?继续往下看。<br />
(再次提醒大家注意questions和queries的区别)</p>
<h1 id="dtq11-15">查询的总体分布报表(DTQ):第11-15行</h1>
<p>所有的请求都可以被粗略的分入五类:数据操作语句(DMS),查询缓存命中(QC Hits),COM_QUIT,其他的COM_命令以及其他未知的东东。接下来的五行分别显示这些,从大到小排列。这样你可以一眼看出mysql最重要的任务是什么了。一般的说,DMS和QCHits是主角,COM_是必须的,额,路人甲……</p>
<p>再详细解释每行之前,提示一下,第三列的比值分母是上面那行的总值。比如本例中DMS占到了82.84%就挺不错的。</p>
<p>DMS包括:select/insert/replace/update/delete(其他的比较偏门,mysqlreport干脆就排除他们了)。正常的说,mysql最应该做的事情,就是这些DMS了。详细内容见17-22行。</p>
<p><code class="highlighter-rouge">QC Hits</code>是mysql从查询缓存里直接获取结果的数量。我们梦寐以求的就是让这个命中率变高,因为这意味着mysql响应变的相当快。但这也意味着你必须接受一定的数据差异性。详细理由见QC报表中的insert/prune和hit/insert比值部分。</p>
<p>在本例中QC Hits达到了16.91%,看来很不错的样子。可别被这么一条数据迷惑了,38-45行的QC报表会给你一个完全不一样的结论。</p>
<p>COM_QUIT,嗯,凑数的。</p>
<p>COM_,如果这个值比较高的话,或许会有些问题,详见23-26行。</p>
<p>Unknown,理想情况下,有上面四个类别就够了。因为有时候,mysql处理了几个请求,却没有记录相应的操作数。所以Unknown有+-两种。+说明mysqlreport统计的多了,-就是少了。这个类别浮动性很大,某些特定情况下,可能会排名很靠前,不过最好还是在最底下吧。</p>
<h1 id="section-8">慢查询:第16行</h1>
<p>第16行非常重要,展示的是mysql执行的慢查询数。默认情况下,配置的<code class="highlighter-rouge">long_query_time</code>是10秒钟。事实上,大家都觉得这太长了,一般都改成1秒,甚至更短,mysql在版本5以后,支持到微妙us级别的。</p>
<p>配置的<code class="highlighter-rouge">long_query_time</code>会在’Slow’后面显示,默认8个字符,所以如果配置的是’9.999999 ms’,只会显示成’999.999 ‘,不过应该不会这么无聊吧~</p>
<p>理想情况下最好这里永远都是0,不过不太可能,多少还是有点。只要第三列的比例低于0.05%就行。</p>
<p>第四列,DMS中的slow比值;第五列,慢查询日志是否记录。强烈建议选择ON。</p>
<h1 id="dms17-22">DMS:17-22行</h1>
<p>和DTQ一样,第一行也是总值,其余内容也是动态排名。这部分内容可以解释mysql服务器偏重于什么类型:select还是insert,或者其他。一般来说会是select吧。明确这个问题,对我们理解其他数值有很大的帮助。比如说:一个insert型的mysql,他的写比率接近1.0,同时带来比较高的表锁。然后很可能用innodb表;一个select型的mysql则相反,读比率高,表锁少,而且很可能用的是MyISAM表。</p>
<p>本例就是一个select型的。65.72%的总请求是select(在DMS的比例提高到79.33%),显然我们可以朝着select的方向进行优化了。</p>
<h1 id="com23-26">COM_:23-26行</h1>
<p>这部分内容都很直观,在mysql协议里都有,像 <code class="highlighter-rouge">com_change_db</code> ,一眼就知道是干嘛的。</p>
<p>如果在DTQ排名里COM_比较高,那说明mysql忙着干自己的事情而不是响应SQL查询。比如说,如果一台mysql的 <code class="highlighter-rouge">com_rollback</code> 高,可能糟糕了,你的事物回滚失败了。结合之前的DTQ报表分析这个东西吧~<br />
一般这些东西不能出什么问题,不过时不时看一眼还是有必要的。</p>
<h1 id="selectsort28-36">select/sort报表:28-36行</h1>
<p>select和sort都是select_的内容。其中最主要的是29和31行:scan和full join。scan展示的是对全表进行扫描的select语句个数。full join和scan很像,除了它还出现于多表查询。程序联合多表进行全表扫描,听起来就慢的可怕……总之,对于这两个数值,只有更低,没有最低!</p>
<h1 id="qc38-45">QC报表:38-45行</h1>
<p>只有mysql版本支持QC并且my.cnf开启了QC的时候该报表才会出现。</p>
<h1 id="section-9">内存使用:39行</h1>
<p>如果内存使用的接近最大设置值,在更下面的Prune数据上也会有反应,因为QC里的查询会被踢出来。</p>
<h1 id="section-10">内存碎片:40行</h1>
<p>内存块碎片(block fragmnt)的详细解释参见《MYSQL手册》的5.14.3章节所述:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>`query_cache_min_res_unit` 的默认值是4KB,大多数情况下足够用了。如果你有一个返回超小结果的海量查询,默认的块大小(即4KB)可能会导致大量的内存碎片,同样也浪费了很多空闲内存块。因为内存不足,碎片会强制删除(prune,我也不知道为啥不叫delete或者purge)QC里的部分内容。这时候你就得降低这个 `query_cache_min_res_unit` 设置。至于空闲块和QC的删除阀值,分别由 `Qcache_free_blocks` 和 `Qcache_lowmem_prunes` 定义。
</code></pre>
</div>
<p>内存碎片的计算方法是空闲内存块除以总内存块数。比值越大,碎片越多,10-20%就已经超出平均水平了。</p>
<p>本例的13.05%还是可以接受的,不过最好还是检查一下 <code class="highlighter-rouge">query_cache_min_res_unit</code> 是不是能调调~</p>
<h1 id="hitsinsertsprunes41-43">Hits/Inserts/Prunes:41-43行</h1>
<p>Hits是最重要的,它反映了select有多少是从QC里获得的应答,当然越多越好。至于insert和prune,或许从44行的比值中更好理解一下。之前有提到prune多说明qc太小,当然这只是一种可能而已。</p>
<p>本例中,只有55%的QC被利用,而碎片又不是太高。prune达到每秒16次,比QCHits高了一倍!打个不太恰当的(这话我加的,感觉比直接理解技术更难懂的比喻)比方:这台mysql的QC就像苹果树一样,苹果还没摘呢,树枝已经被砍掉了……</p>
<h1 id="insertprunehitinsert44-45">insert/prune和hit/insert比:44-45行</h1>
<p>insert/prune是一个波动性的QC指标。一个稳定运行中的QC,insert进QC的查询数量应该大于prune掉的查询数量。而一个不稳定的QC,比值或许是1:1,甚至偏向prune。这说明两个问题:1、QC大小不够;2、mysql试图缓存一切,结果帮了倒忙~</p>
<p>如果是第一种情况,简单的加大QC大小就够了。然后再观察碎片和内存使用率的情况。</p>
<p>但更多的时候是第二种情况。因为QC设置里开启的默认type1就是要求mysql尽可能的缓存一切东西。</p>
<p>mysql官方说明里这么解释这个<code class="highlighter-rouge">type 1</code>的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>“缓存一切查询结果,除非查询时使用'select sql_no_cache'方式”。
</code></pre>
</div>
<p>可惜这个 <code class="highlighter-rouge">sql_no_cache</code> 基本没人用。另一个稍微好一些的方式是 <code class="highlighter-rouge">type 2 demand</code> ,解释如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>“只有在查询使用 `select sql_cache` 时才缓存查询结果”。
</code></pre>
</div>
<p>这个type对开发人员要求比较多,因为他们得明确指出哪些要缓存,哪些不要。不过也没人比他们更清楚到底哪些是该缓存的啦~</p>
<p>hit/insert用来反映QC的有效性。理想情况是:mysql插入一批稳定的查询到QC里,然后源源不断的命中这批结果……所以,如果QC的有效性足够,这个比值应该是偏向hit的。如果不幸的偏向了insert,那说明QC其实没起到太大的作用。比如说1:1,一次insert用了一次hit,然后就被替换了,这完全违背了使用QC的初衷。不过还有更糟的,比如0.34:1,一次都没用上,就被prune掉了……</p>
<p>本例的QCHits不低,hit/insert却不高。再考虑到内存使用和碎片情况也还可以。或许真的有必要换成type2的DEMAND了~~</p>
<h1 id="section-11">表锁报表:47-49行</h1>
<p>表锁报表包括两行,一行是总数,一行是当前数。锁等待对于数据库来说永远是糟糕的事情。第三列的总比值反应了一个综述的情况,无论如何不能高过10%,否则肯定就带来一大堆的索引和慢查询问题!</p>
<h1 id="section-12">表报表:51-53行</h1>
<p>也是两行,一行是当前mysql打开表的个数、表缓存的使用率,一行是mysql运行以来的平均值。</p>
<p>这里有两个值比较重要。一个是表缓存使用率,哪怕高到100%都行。不过要是真高到100%了,可能你的’table_cache’设置已经不够了,赶紧加大吧。第二个是当前打开表的比率,这个也能协助判断’table_cache’设置是否合理。一般这个值应该小于每秒1次。不过一个负载比较高而又运行的还不错的mysql,可能能达到每秒打开7次表,依然保持100%的表缓存~</p>
<h1 id="section-13">连接数报表:55-57行</h1>
<p>如果最大连接数曾经接近过100%,请加大’max_connection’设置。不过事实上,默认的100已经足够绝大多数哪怕相当繁忙的mysql使用了,盲目加大这个设置其实不对。一个mysql链接持续1秒钟,100个就是足足100秒。所以如果连接数太高,或者说一直在慢慢涨,问题很可能在别的地方,比如慢查询、糟糕的索引、甚至DNS解析太慢。在修改这个数的事情,还是先去研究一下为什么100个还不够呢?</p>
<p>至于每秒连接数,只要mysql运行正常,高低无所谓。不过大多数mysql这个值在每秒5次以下~~</p>
<h1 id="section-14">临时表报表:59-62行</h1>
<p>mysql可以在内存、磁盘甚至临时文件上创建临时表。这三种情况分别对应下面的三条报告。这些数据没有标准值,不过必须要知道的是磁盘上的临时表示最慢的。mysql一般也避免在磁盘上创建临时表,除非达到了’tmp_table_size’的阀值。这个阀值会显示在内存(Table)那行的Size列后面。至于内存和临时文件的数值到底该多少,取决于mysql数据库的硬件配置。</p>
<h1 id="section-15">线程、中断、流量报表:64-76行</h1>
<p>这三个报表是最重要的,系统级别的问题不用多说了都……</p>
<p>这里面有一个需要注意的地方:66行的线程命中率。每个mysql的连接都是一个单独的线程。mysql启动时,只创建不多的几个线程和一个线程缓存,以节省不断创建和销毁线程的开销,哪怕这个开销不怎么明显。当mysql的连接数超过了线程缓存数(由 <code class="highlighter-rouge">thread_cache_size</code> 定义)时,MySQL开始出现线程抖动(‘thread thrash’)。为了接纳新的连接,mysql疯狂的创建新线程,结果自然是线程命中率大幅下滑。</p>
<p>线程抖动是问题么?Yahoo的Jeremy Zawondy在博客中写了如下一段话:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>所以上面这个故事的教训就是:如果你的服务器老是接受一些快速连接,想办法加大你的线程缓存吧,直到你的`show status`命令里`threads_created`不再飙升为止!你的CPU绝对会感谢你的。线程缓存不是啥大问题。可是你解决完真的大问题后,它就是最大的问题了……(汗)
</code></pre>
</div>
<p>所以,如果真出现线程抖动的话,加大’thread_cache_size’吧。</p>
<p>比如本例,线程缓存命中率只有可怜的0.05%!基本意味着每个连接都是新创建了线程。不过看看报表其他的内容,这也是必然的:缓存里一个线程都没有(66行),201个线程被创建(67行),202个连接(57行)……</p>
<h1 id="innodb78-92">innodb缓冲池报表:78-92行</h1>
<p>mysql版本5.0.2之后才在<code class="highlighter-rouge">show status</code>里加入了innodb_内容。所以这部分报表只在该版本之后有效。</p>
<p>innodb存储引擎的重要特性就是它把表数据和索引都缓存在缓冲池里。缓冲池内的页大小是16KB,可以包含各种不同的数据类型。本报表就展示了这些页的数值。</p>
<p>注:因为mysqlreport比较老,对innodb的采集分析不如myisam多。</p>
<h1 id="section-16">使用率:79行</h1>
<p>这个可以类比第4行的 <code class="highlighter-rouge">key buffer used</code>。不过myisam引擎只缓存索引,而innodb也存数据。所以要想知道这个使用率的详细情况,还得看81-85行的内容。</p>
<p>显然,必须避免mysql运行到缓冲池溢出的地步。myisam溢出,只会导致性能下降(索引读写变慢),而innodb的溢出问题就多了,因为几乎所有东西都依赖缓冲池。所以最好还是配置好自增长缓冲池(‘auto-extending buffer pool’)吧。</p>
<h1 id="section-17">读命中:80行</h1>
<p>同样跟地7行的<code class="highlighter-rouge">key read hit</code>类似。不过对innodb来说这个数值更重要。缓冲池显示的是从内存读取的比例,所以一般都接近100%,绝大多数情况至少大于99.98%。</p>
<h1 id="section-18">页:81-85行</h1>
<p>各种各样的关于缓冲池中页的指标。比如82行的空闲页、83行的数据页、84行的杂页、85行的锁定页。</p>
<p>空闲页就是79行使用率的对立方。</p>
<p>数据页和空闲页一样是自描述的。我们没法知道这些页里有哪些数据类型。本行还有一列%Dirty,展示已经被修改过,但还没有被刷新到磁盘存储的数据页的比率。</p>
<p>另两样就更没什么可说的了。mysql手册上只是简单的这么描述这两样页:“用于管理分配行锁和自适应哈希索引导致的开销使用的页”和“目前正在读写、或者因为其他原因无法被刷新的页”。</p>
<h1 id="section-19">读:86-89行</h1>
<p>接下来的四行是关于innodb缓冲池的读情况的。86行是从内存读取的数量。在一个比较繁忙的innoDB引擎mysql服务器上,这个值应该非常大,因为innodb总是试图从内存里的缓冲池读取页。这个数值可以用来衡量innodb缓冲池的吞吐量。因为几乎所有inondb需要的东西都是在缓冲池里,所以缓冲池的读性能是越快越好。哪怕超过每秒200000次也不是不可能的。</p>
<p>87行的数值就小的多了。官方解释叫做“innodb无法从缓冲池获得而只能进行单页读取的逻辑读操作数”,其实就是从磁盘读的数量。</p>
<p>88行,随机读。官方解释“innodb启动的随机读取数。只有对表的大部分内容进行随机扫描的时候才会出现”。</p>
<p>89行,顺序读。官方解释“innodb启动的顺序读取数。只有进行全表扫描的时候才会出现”。全表扫描可不是什么好事,这个数值还是小点好。</p>
<h1 id="section-20">写:90行</h1>
<p>和86行类似,也可以用来衡量innodb的吞吐量。本行显示写的数量以及读写的比率。如果服务器主要操作是update和insert的话,这个值也会比较高。</p>
<h1 id="section-21">刷新:91行</h1>
<p>缓冲池的页刷新请求数。</p>
<h1 id="section-22">空闲等待:92行</h1>
<p>mysql手册上这么描述这个变量:<br />
一般情况下,innodb缓冲池的写操作是后台运行的。不过,如果出现必须要读写一个页可偏偏没有可用的新页时,(innodb)就只能先等待页的刷新了。这个变量就是这些等待的总数。只要缓冲池的大小设置得当,等待数应该会很小。</p>
<h1 id="innodb94-100">innodb锁报表:94-100行</h1>
<p>innodb的行锁变量是mysql5.0.3之后加入的。MyISAM引擎是表锁,而innoDB是行锁。所以当你使用innodb时这几个变量的值非常重要。</p>
<h1 id="section-23">等待:95行</h1>
<p>“等待某行解锁的累积次数”,最好是0次。</p>
<h1 id="section-24">当前:96行</h1>
<p>“当前正在等待解锁的行个数”,最好是0次。</p>
<h1 id="section-25">时间分析:97-100行</h1>
<p>98-100行显示了毫秒(ms)级行锁等待数据。分别是总值、平均值和最大值。同样最好是0次。</p>
<h1 id="innodb102-121">innoDB数据、页、行报表:102-121行</h1>
<p>这部分报告,一般广泛的用于衡量innodb引擎的吞吐量指标。</p>
<h1 id="section-26">数据:103-110行</h1>
<p>第一部分,数据,列出了四种类型的操作:读、写、刷新(fsync)和等待(pending)。</p>
<ol>
<li>第一个类型:读,指的是整个innodb引擎完成所有的数据读取次数——注意:不是整个数据读取字节数或者类型,而是innodb完成的数据读取次数。</li>
<li>第二个类型:写,和读一样也是次数的统计。</li>
<li>第三个类型:刷新,同样的,innodb从内存写入磁盘的次数。这个值应该会比前两个小。</li>
<li>第四个类型:等待,又被分成了三行(108-110),分别是读、写、刷新的等待次数。</li>
</ol>
<h1 id="section-27">页:112-115行</h1>
<p>这部分包括三种自描述类型:创建、读取、写入,分别用来表示缓冲池中页的创建、读取和写入的数量和速率(即每秒操作数)。由于没有指出具体是哪种页,这几个数值也就是用来概述一下innodb引擎的吞吐量罢了。</p>
<h1 id="section-28">行:117-121行</h1>
<p>最后一部分,行,只是一些比较泛泛的数值。包括对行的delete/insert/read/update操作。这些数值会比较大,而速率部分同样也是些概述参考……</p>
<h1 id="section-29">结论</h1>
<p>现在我们已经阅读甚至分析完了整个报表,可以对本例mysql做出一个总体的评价了。</p>
<p>总的来说,这个服务器运行情况还是不错的,几个关键指标都很好:key buffer只用了12%,DMS和QCHits占了请求的99%,也没有什么Com_问题,表锁也不错,表缓存只用了10%,连接数很低。</p>
<p>至于innodb引擎,服务器也在使用,但比重不大。目前这些innodb的状态变量反映出的情况看,一些正常。</p>
<p>不过还是有些优化的余地。首先一点,也是最重要的一点,就是查询缓存QC不太稳定;然后是线程缓存,建议调高 <code class="highlighter-rouge">thread_cache_size</code>,知道线程缓存命中率回升到满意值~~</p>
日志计算(awk进阶)
2011-01-07T00:00:00+08:00
bash
awk
http://chenlinux.com/2011/01/07/compute-flow-from-access_log-by-awk-2
<table>
<tbody>
<tr>
<td>曾经用awk写过一个日志流量计算的单行命令。因为awk中没有sort函数,所以在中途采用了</td>
<td>sort</td>
<td>的方式,导致效率很低。在计算50GB+的日志时,运算时间慢的不可忍受。</td>
</tr>
</tbody>
</table>
<p>设想了很多方法来加快运算,比如舍弃awk改用perl来完成,如下:<br />
<code class="highlighter-rouge">perl#!/usr/bin/perl
use warnings;
use strict;
my $access_log = $ARGV[0];
my $log_pattern = qr'^.*?:(\d\d:\d\d):(\S+\s){5}\d+\s(\d+).+';
my %flow;
my ($traffic, $result) = (0, 0);
open FH,"< $access_log" or die "Cannot open access_log";
while (defined(my $log = <FH>)) {
$flow{$1} += $3 if $log =~ /$log_pattern/;
#print $1." ".$3." ".$flow{$1} * 8 / 300 / 1024 / 1024,"\n";
}
close FH;
foreach my $key ( sort keys %flow ) {
my $minute = $1 if $key =~ /\d\d:\d(\d)/;
$traffic += $flow{$key};
if ( $minute == '0' or $minute == '5' ) {
$result = $traffic if $traffic > $result;
$traffic = '0';
}
}
print $result * 8 / 300 / 1024 / 1024;
</code><br />
好吧,这个正则太过垃圾,请无视,但至少在管道和系统sort上浪费的时间还是大大的节省了的。</p>
<p>然后在CU上翻到一个老帖子,提供一个比较不错的awk思路,命令如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>awk <span class="s1">'!b[substr($4,14,5)]++{print v,a[v]}{v=substr($4,14,5);a[v]+=$10}END{print v,a[v]}'</span>
</code></pre>
</div>
<p>这里用substr()跟指定FS相比那个效率高未知,不过采用!b++的方式来判断某时间刻度结束,输出该时刻总和,在顺序输入日志的前提下,运算速度极快(就剩下一个加法和赋值了)。</p>
<p>注意:此处b[]内不能偷懒写v,!b[v]++永远只会输出时刻的第一行数值。</p>
<p>不过真实使用时不尽如人意。逻辑上推导了很久没发现问题,结果在重新运行前面的perl时看了看while中的print,发现这个日志因为是从各节点合并出来的日志,其时间并不是顺序排列的!!</p>
<p>另,刚知道gawk3.1以上提供了asort()和asorti()函数,可以研究一下~</p>
<p>采用time命令测试一下<br />
<code class="highlighter-rouge">bash
gawk '{a[substr($4,14,5)]+=$10}END{n=asorti(a,b);for(i=1;i<=n;i++){print b[i],a[b[i]]*8/60/1024/1024}}' example.com_log | awk '{if($2>a){a=$2;b=$1}}END{print b,a}'
</code></p>
<p>一个35G的日志文件只用了6分钟。</p>
<p>然后更简单的<br />
<code class="highlighter-rouge">bash
gawk '{a[substr($4,14,5)]+=$10}END{n=asort(a);print a[n]*8/60/1024/1024}' example.com_log
</code></p>
<p>不过简单的运行发现只比前一种快不到10秒钟。而前一种还能输出峰值的时间,可读性更好一些~</p>
resin与ipv6
2010-12-30T00:00:00+08:00
web
resin
http://chenlinux.com/2010/12/30/resin-ipv6
<p>一台nginx+resin的应用服务器出现大量的ipv6下的CLOSE_WAIT。重启后十五分钟就累积到了1500+。</p>
<p>调整resin的keepalive-timeout和nginx的proxy_read_timeout,没有丝毫改变。决定先关掉ipv6。</p>
<p>如果要关掉整个linux的ipv6,需要修改/etc/modprobe.conf然后reboot,显然没法在生产环境上直接操作,只能寻求应用监听上的办法。</p>
<p>修改sysctl -w net.ipv6.bindv6only=1,然后重启应用,赫然发现resin重启失败,只有perl进程,没有java进程了。改回0,java立刻启动成功。</p>
<p>google了一下,在<a href="http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=560044">http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=560044</a>中看到了答案——原来java默认就是用ipv6。</p>
<p>增加resin/bin/wrapper.pl相关行如下:</p>
<p>$EXTRA_JAVA_ARGS=”-Djava.util.logging.manager=com.caucho.log.LogManagerImpl”;<br />
$EXTRA_JAVA_ARGS.=” -Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl”;<br />
$EXTRA_JAVA_ARGS.=” -Djava.net.preferIPv4Stack=true”;#新增参数</p>
<p>重启,就只在ipv4上了。</p>
TCP响应时间监测
2010-12-28T00:00:00+08:00
monitor
gnuplot
http://chenlinux.com/2010/12/28/intro-tcprstat
<p>对于squid等服务器,其日志中就含有响应时间。但是,这个时间只是服务器软件处理过程的时间,进程一旦交出去,在网卡等处的时间,它就管不着了。而percona出品一款迷你型小工具,叫做tcprstat,正好派上用场~</p>
<p>tcprstat下载地址见:<a href="http://www.percona.com/docs/wiki/tcprstat:start">http://www.percona.com/docs/wiki/tcprstat:start</a></p>
<p>本来是percona用来监测mysql响应时间的。不过对于任何运行在TCP协议上的响应时间,都可以用。不过据我试验,(至少非编译的64位二进制版如此)这个工具对linux内核版本也是有要求的,我在RH的AS4上运行就提示kernel不够……</p>
<p>对于使用一个2000行代码的工具,网页上的说明已经相当清晰。直接开用吧:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget http://github.com/downloads/Lowercases/tcprstat/tcprstat-static.v0.3.1.x86_64 --no-check-certificate -O /sbin/tcprstat
chmod +x /sbin/tcprstat
tcprstat -p 1521 -t 10 -n 0 -f <span class="s1">'%T\t%n\t%M\t%a\t%95M\t%99M\n'</span>
timestamp count max avg 95_max 99_max
1293528181 339 4429229 142446 617688 2196833
</code></pre>
</div>
<p>That’s all!</p>
<p>监听oracle的1521端口,每10秒一次统计,长期运行,输出格式为“UNIX时间,响应个数,最长响应时间,平均响应时间,95%响应时间,99%响应时间”(这个%可以自定义数值)</p>
<p><strong>注意:这个响应时间是microsecond,即us,等于0.000001s。</strong></p>
<p>突然想起来基调的smoke图。用这个输出数据给gnuplot,相当容易画出类似的效果~给不同%的数据定义渐进颜色画柱状图(注意叠加次序),avg画连线图,count或许可以画在top的x轴上~先贴个简易版的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">set</span> terminal png size 550,350 <span class="se">\
</span>xffffff x000000 x404040 <span class="se">\
</span>xcfcfcf x8a8a8a x4a4a4a x00ff00
<span class="k">set</span> output <span class="s2">"log.png"</span>
<span class="k">set</span> autoscale
<span class="k">set</span> xdata time
<span class="k">set</span> timefmt <span class="s2">"%s"</span>
<span class="k">set</span> xlabel <span class="s2">"time"</span>
<span class="k">set</span> ylabel <span class="s2">"microsecond"</span>
<span class="k">set</span> title <span class="s2">"Oracle response time"</span>
<span class="k">set</span> grid
plot <span class="s2">"tcp.log"</span> using 1:3 title 'max' with filledcurves x1, <span class="se">\
</span><span class="s2">"tcp.log"</span> using 1:6 title '97%' with filledcurves x1, <span class="se">\
</span><span class="s2">"tcp.log"</span> using 1:5 title '90%' with filledcurves x1, <span class="se">\
</span><span class="s2">"tcp.log"</span> using 1:4 title 'avg' with lines
</code></pre>
</div>
<p>效果如下:</p>
<p><img src="/images/uploads/log-2.png" alt="tcprstat-gnuplot" /></p>
ims在nginx上的处理(无责任猜测)
2010-12-23T00:00:00+08:00
nginx
http://chenlinux.com/2010/12/23/if-modified-since-process-in-nginx
<p>最近得知CDN方面默认配置了reload-into-ims,而我们的html因为采用了ssi的include方式的原因,是没有last-modified的。在这种情况下的处理结果,让人好奇~</p>
<p>因为手头没测试机器,仅从我一知半解的nginx代码上推测一下:</p>
<p>1、之前博客里已经写过,nginx的ssi_module,在ngx_ssi_header_filter中简单的采用了ngx_http_clear_last_modified(r)抹去了last-modified的输出。</p>
<p>2、在nginx的module定义中,各filter的顺序如下:<br />
<code>ngx_module_t *ngx_modules[] = {</code><code>
</code><code>...............................................
</code><code>&ngx_http_write_filter_module,</code><code>
</code><code>&ngx_http_header_filter_module,</code><code>
</code><code>&ngx_http_chunked_filter_module,</code><code>
</code><code>&ngx_http_range_header_filter_module,</code><code>
</code><code>&ngx_http_gzip_filter_module,</code><code>
</code><code>&ngx_http_postpone_filter_module,</code><code>
</code><code>&ngx_http_ssi_filter_module,</code><code>
</code><code>&ngx_http_charset_filter_module,</code><code>
</code><code>&ngx_http_userid_filter_module,</code><code>
</code><code>&ngx_http_headers_filter_module,</code><code>
</code><code>&ngx_http_copy_filter_module,</code><code>
</code><code>&ngx_http_range_body_filter_module,</code><code>
</code><code>&ngx_http_not_modified_filter_module,</code><code>
</code><code>NULL</code><code>};</code><br />
越后面的越先处理。也就是说,一个带有ims请求,会先经过not_modified_filter,然后才是ssi_filter。<br />
而在ssi抹掉last-modified之前,文件是应该存在last-modified的。<br />
nginx默认情况下,对一个ims请求的处理流程,参见淘宝核心系统部雕梁童鞋的博客《<a href="http://www.pagefault.info/?p=66">nginx中处理http header详解(1)</a>》。大体上,是nginx获得这个请求文件的Mtime,存为变量last_modified_time;然后通过ngx_http_parse_time()换算IMS中的时间。如果ims!=last_modified_time,读取文件内容成response-body,否则clear掉content-length/content-type/content-encoding/accept-ranges等,进入下一步。</p>
<p>那么,如果一个html页本身没有修改,其中含有<!--#include virtual="/ssi/test.html"-->,而这个/ssi/test.html却变动了,那么在向这个html页发送ims的时候,就应该会是返回一个Not Modified,但在ssi_filter里又把last-modified也clear了……</p>
<p>3、在ssi_filter里如果要输出last-modified的话,如果单纯只是注释掉clear,并不起作用。不过在ngx_http_ssi_filter_module.c中,看到ngx_http_ssi_include()中调用了ngx_http_ssi_stub_output()这个handler,对virtual或file的拼接时处理header中的content-type如下:<br />
if (!r->header_sent) {<br />
r->headers_out.content_type_len =<br />
r->parent->headers_out.content_type_len;<br />
r->headers_out.content_type = r->parent->headers_out.content_type;<br />
if (ngx_http_send_header(r) == NGX_ERROR) {<br />
return NGX_ERROR;<br />
}<br />
}<br />
return ngx_http_output_filter(r, out);</p>
<p>或许在这里加上<br />
对r->parent->headers_out.last_modified_time和r->child->headers_out.last_modified_time的大小判断,然后赋值给r->headers_out.last_modified_time,然后把ssi_filter优先到not_modified_filter之前来?</p>
<p>C盲睡觉去也~~找时间写几个shtml测一下就知道自己对next_header_filter的理解对不对了~</p>
wget和curl测试时的小区别
2010-12-23T00:00:00+08:00
linux
http://chenlinux.com/2010/12/23/diff-between-wget-curl
<p>在对网站内容是否更新进行测试时,最常用的两个工具就是wget和curl。不过两个工具之间还是有一些小区别,甚至很可能影响到测试结论的。记录一下:</p>
<p>1、在查看response-header的时候,我们习惯用的是wget -S和curl -I,但是:wget -S的时候,发送的是GET请求,而curl -I发送的是HEAD请求。如果测试url此时没有被缓存过,直接使用curl -I进行测试,永远都会返回MISS状态。所以最好先wget一次,再用curl -I。</p>
<p>2、在查看下载速度时,常常发现wget和curl耗时差距较大。因为wget默认使用HTTP/1.0协议,不显式指定–header=”Accept-Encoding: gzip,deflate”的情况下,传输的是未经压缩的文件。而curl使用HTTP/1.1协议,默认接受就是压缩格式。</p>
<p>3、在测试缓存层配置时,有时发现wget可以HIT的东西,curl却始终MISS。对此可以开启debug模式进行观察跟踪。<br />
wget自带有-d参数,直接显示request-header;curl只有-D参数,在采用GET请求的时候,将response-header另存成文件,所以只好在squid上debug请求处理流程(当然也可以去网络抓包),结果发现,curl的GET请求,都带有”Pragma: no-cache”!而wget需要另行指定–no-cache才会。按照squid的默认配置,对client_no_cache是透传的,所以curl永远MISS,除非squid上配置了ignore-reload/reload-into-ims两个参数,才可能强制HIT。</p>
通过snmp协议监控NetApp
2010-12-22T00:00:00+08:00
monitor
snmp
NetApp
http://chenlinux.com/2010/12/22/monitor-netapp-by-net-snmp
<p>NetApp作为专业存储,用起来还是比较让人放心的,不过放心不代表放手不管,一些重要的监控还是要做的。比如基本的CPU负载、网卡流量、磁盘使用率,作为数据存储特别关注的IOPS、DiskIO(因为有cache的原因,所以NetIO和DiskIO是不同时的,单从网卡进出不能判定磁盘的真实读写)。</p>
<p>从google中找到了NetApp的MIBtree,见<a href="https://support.ipmonitor.com/mibs/NETWORK-APPLIANCE-MIB/tree.aspx">https://support.ipmonitor.com/mibs/NETWORK-APPLIANCE-MIB/tree.aspx</a>或<a href="http://www.mibdepot.com/cgi-bin/getmib3.cgi?abc=0&n=NETWORK-APPLIANCE-MIB&r=netapp&f=netapp_1_4.mib&t=tree&v=v1&i=0&obj=cp">http://www.mibdepot.com/cgi-bin/getmib3.cgi?abc=0&n=NETWORK-APPLIANCE-MIB&r=netapp&f=netapp_1_4.mib&t=tree&v=v1&i=0&obj=cp</a></p>
<p>(多贴一个,省的万一哪个站挂了……)</p>
<p>CPU等项没什么问题,问题在于IO的获取。<br />
与普通服务器的流量一样,很明显的看到了miscNfsOps、miscNetRcvdKB和miscNetSentKB三个值。我们可以很简单的通过后个值的差值运算出速率。但如果单取一个值的时候,会发现一个很诡异的情况,见下:</p>
<h1 id="snmpwalk--v-1--c-public-101010118-1361417891223--awk-print-nf">snmpwalk -v 1 -c public 10.10.10.118 .1.3.6.1.4.1.789.1.2.2.3 | awk ‘{print $NF}’</h1>
<p>-1948753579</p>
<p>居然是个负值!</p>
<p>难道是对这个MIB的理解有问题?可通过如下命令计算的结果,和通过交换机端口获取的流量却基本符合了:</p>
<p>a=<code class="highlighter-rouge">snmpwalk -v 1 -c public 10.10.10.118 .1.3.6.1.4.1.789.1.2.2.3 | awk '{print $NF}'</code>;sleep 10;snmpwalk -v 1 -c public 10.10.10.118 .1.3.6.1.4.1.789.1.2.2.3 | awk ‘{print ($NF-“‘$a’”)*8/10”Kbps”}’<br />
12364.8Kbps</p>
<p>虽然数值都变负了,但差量还是对的……汗,如果通过AVERAGE方式取这个差值绘图,倒不要紧,如果通过普通的流量Counter方式,这个图全反过X轴去了应该~</p>
<p>这个情况很容易让我想到用cacti获取网卡流量时的配置,如果选用默认的32bits绘图时,在流量较大时也会出现这类负值或者干脆画不出来的情况。</p>
<p>其次,DiskIO没有和Net一样的MIB;但net、disk和ops都有一对miscHigh<strong><em>/miscLow</em></strong>的数值。</p>
<p>这对值怎么用?MIB上的解释看起来超级茫然:<br />
miscLowNfsOps:The total number of Server side NFS calls since the last boot. This object returns the least significant 32 bits of the value.<br />
miscHighNfsOps:The total number of Server side NFS calls since the last boot. This object returns the most significant 32 bits of the value.</p>
<p>通过度娘了解了一下least significant bit和most significant bit的概念,原来LSB和MSB是在底层开发时的概念,因为2进制的字串比较长,所以通过MSB和LSB的命名方式来标明字串的高位(普通PC和MAC在存数据的时候顺序是反的,所以必须标记,还有其他原因,比如用MSB的1、0来表示正负数等)</p>
<p>那么这个most significant 32bits也就理解了~~就是从高位开始往低算的32位。least反过来……合并起来就是64位的完成数据了……2进制数的合并,也就是说用$MSBs * 2**32 + $LSBs。My God~</p>
<p>最后在zenoss的maillist上看到了相同的问题,其中网友的回答是:NetApp使用的snmp协议是v1版本,无法直接提供64bits的计数,只能变通一下,改用这种拆分方式了。</p>
<p>最后,举例一个监控ops的perl脚本。原脚本是centreon项目提供的,删除了一些和ops无关的语句。从中也可以学到Net::SNMP模块的使用,hash的解引用等~<br />
<code class="highlighter-rouge">perl#!/usr/bin/perl -w
use strict;
use Net::SNMP;
use Getopt::Long;
use lib "/usr/local/nagios/libexec";
use utils qw(%ERRORS $TIMEOUT);
my $o_host = undef; # hostname
my $o_community = undef; # community
my $o_port = 161; # port
my $o_warn = undef; # warning limit
my $o_crit= undef; # critical limit
my $o_timeout= 10;
my $exit_code = undef;
my $o_type=undef;
my $output=undef;
my $o_perf= undef;
my %oids = (
'cpuUsage' => ".1.3.6.1.4.1.789.1.2.1.3.0",
'globalStatus' => ".1.3.6.1.4.1.789.1.2.2.4.0",
'nfsHighOps' => ".1.3.6.1.4.1.789.1.2.2.5.0",
'nfsLowOps' => ".1.3.6.1.4.1.789.1.2.2.6.0",
'netRecHighBytes' => ".1.3.6.1.4.1.789.1.2.2.11.0",
'netRecLowBytes' => ".1.3.6.1.4.1.789.1.2.2.12.0",
'netSentHighBytes' => ".1.3.6.1.4.1.789.1.2.2.13.0",
'netSentLowBytes' => ".1.3.6.1.4.1.789.1.2.2.14.0",
'diskReadHighBytes' => ".1.3.6.1.4.1.789.1.2.2.15.0",
'diskReadLowBytes' => ".1.3.6.1.4.1.789.1.2.2.16.0",
'diskWriteHighBytes' => ".1.3.6.1.4.1.789.1.2.2.17.0",
'diskWriteLowBytes' => ".1.3.6.1.4.1.789.1.2.2.18.0",
);
my @oidlist=($oids{nfsHighOps},$oids{nfsLowOps});
sub check_options {
Getopt::Long::Configure ("bundling");
GetOptions(
'H:s' => \$o_host, 'hostname:s' => \$o_host,
'p:i' => \$o_port, 'port:i' => \$o_port,
'C:s' => \$o_community, 'community:s' => \$o_community,
'c:s' => \$o_crit, 'critical:s' => \$o_crit,
'w:s' => \$o_warn, 'warn:s' => \$o_warn,
# 'T:s' => \$o_type,
);
}
########## MAIN #######
check_options();
# Connect to host
my ($session,$error);
#关键点:创建一个session连接被监控主机
($session, $error) = Net::SNMP->session(
-hostname => $o_host,
-community => $o_community,
-port => $o_port,
-timeout => $o_timeout
);
if (!defined($session)) {
printf("ERROR: %s.\n", $error);
exit $ERRORS{"UNKNOWN"};
}
my $resultat=undef;
# Get rid of UTF8 translation in case of accentuated caracters (thanks to Dimo Velev).
#这里没太看懂perldoc,猜测是不开启MIB和oid的转换,这样输出结果比较简洁,不过注释掉这句运行结果毫无影响
$session->translate(Net::SNMP->TRANSLATE_NONE);
#取值的关键,get_request返回一个hash的引用。
#perldoc的原文是:“A reference to a hash is returned in blocking mode which contains the contents of the VarBindList. In non-blocking mode, a true value is returned when no error has occurred.”
#get_request()中-callback和-delay是non-blocking模式的,而\@oids是blocking模式。
#也就是说这个脚本里返回的是一个引用。
if (Net::SNMP->VERSION &lt; 4) {
$resultat = $session->get_request(@oidlist);
} else {
$resultat = $session->get_request(-varbindlist => \@oidlist);
}
if (!defined($resultat)) {
printf("ERROR: Description/Type table : %s.\n", $session->error);
$session->close;
exit $ERRORS{"UNKNOWN"};
}
$session->close;
my $new_nfs_ops;
my $left_shift= 2**32;
my $last_nfs_ops = 0;
my $row ;
my $last_check_time ;
my $update_time;
my @last_values=undef;
my $flg_created = 0;
#解引用关键点:对%$resultat的解引用$$resultat{$oids{nfsHighOps}},其中$oids{nfsHighOps}是另一个hash——%oids中的值。
#可以采用foreach my $value ( values %$resultat ) { print "$value\n"; }的方式列出hash中的各个值。
#按照MSBs和LSBs的划分方法,通过2**32的方式合并得到64bits计数。
$new_nfs_ops= $$resultat{$oids{nfsHighOps}} * $left_shift + $$resultat{$oids{nfsLowOps}};
#输出到文本文件,因为是给nagios做监控脚本,所以必须通过差值的方式计算average型数值,而不像cacti绘图时那样可以直接传递counter型数值。
if (-e "/tmp/traffic_ops_".$o_host) {
open(FILE,"&lt;"."/tmp/traffic_ops_".$o_host);
while($row = &lt;FILE>){
@last_values = split(":",$row);
$last_check_time = $last_values[0];
$last_nfs_ops = $last_values[1];
$flg_created = 1;
}
close(FILE);
} else {
$flg_created = 0;
}
$update_time = time();
unless (open(FILE,">"."/tmp/traffic_ops_".$o_host)){
print "Check mod for temporary file : /tmp/traffic_ops_".$o_host. " !\n";
exit $ERRORS{"UNKNOWN"};
}
print FILE "$update_time:$new_nfs_ops:$new_cifs_ops";
close(FILE);
if ($flg_created == 0){
print "First execution : Buffer in creation.... \n";
exit($ERRORS{"UNKNOWN"});
}
my $nfs_diff=$new_nfs_ops - $last_nfs_ops;
$nfs_diff=$new_nfs_ops if ($nfs_diff &lt; 0);
my $time_diff=$update_time - $last_check_time;
$time_diff=$update_time if ($time_diff &lt; 0);
my $nfs_ops = $nfs_diff / ( $time_diff );
printf("Nfs ops : %.2f ops/sec ", $nfs_ops);
printf("|nfsOps=".$nfs_ops."\n");
exit($ERRORS{"OK"});</code></p>
for/while循环的区别
2010-12-16T00:00:00+08:00
bash
http://chenlinux.com/2010/12/16/diff-between-for-while
<p>一般习惯使用for循环,在一年前写cgi的时候,还为这郁闷过一阵:for i in <code class="highlighter-rouge">cat ip</code>时,会自动的把文件中每行内容按照空格分割传递,最后采用先把空格改成+号的方式解决。</p>
<p>今天看CU,发现也有人提出这个问题,而解决办法很简单——用while循环即可。</p>
<table>
<tbody>
<tr>
<td>另,while循环有两个用法,cat a</td>
<td>while read和while;do;done<a,pipe方式的变量,仅在循环内有效,又是一个区别~~</td>
</tr>
</tbody>
</table>
<p>下面是示例:</p>
<p>[root@localhost ~]# cat info<br />
a b c d<br />
[root@localhost ~]# for i in <code class="highlighter-rouge">cat info </code>;do echo $i;done<br />
a<br />
b<br />
c<br />
d<br />
[root@localhost ~]# i=123;while read i;do echo $i;done<info;echo $i<br />
a b c d</p>
<p>[root@localhost ~]# i=12;cat info |while read i;do echo $i;done;echo $i<br />
a b c d<br />
12<br />
[root@localhost ~]#</p>
<p>另,看到一个网站,专门介绍单行shell命令的,对SA来说,比较有用,url如下:<br />
<a href="http://www.commandlinefu.com/commands/browse">http://www.commandlinefu.com/commands/browse</a></p>
weathermap-cacti-plugin学习(3)
2010-12-12T00:00:00+08:00
monitor
cacti
php
http://chenlinux.com/2010/12/12/learning-weathermap-cacti-plugin-3
<p>今天继续啃weathermap的php代码,因为lib的readdata里return了$inbw和$outbw,尝试在之前理解的ReadConfig()相应match处加上了一段 <code class="highlighter-rouge">if($inbw=='0'){$this->width=0;$linematched++;}elseif</code>……等几分钟后cache过期,一看weathermap效果,所有的链路曲线箭头图都变成了一根直线~~然后仔细看了看这串if之前的while,发现原来weathermap不是每次针对数据进行config匹配,而是统一读取一次config。也就是说ReadConfig()里的任何修改都会对全局起作用。</p>
<p>既然在源头的配置参数无法修改,那么就只能在末端的绘图的时候做出改变了,找到 <code class="highlighter-rouge">draw_curve()</code> 函数,这个函数就是画线的。最终由 <code class="highlighter-rouge">Draw()</code> 函数分别调用 <code class="highlighter-rouge">calc_curve</code> 描点,<code class="highlighter-rouge">draw_curve</code> 连线,<code class="highlighter-rouge">drawlabel</code> 画框。<code class="highlighter-rouge">Draw()</code> 内容是一个顺序处理过程,在最后的 <code class="highlighter-rouge">drawlabel()</code> 前面,可以很清晰的看到 <code class="highlighter-rouge">$task[*]</code> 是怎么取出的:<br />
<code class="highlighter-rouge">php
$outbound=array($q1_x,$q1_y,0,0,$this->outpercent,$this->bandwidth_out,$q1_angle);
$inbound=array($q3_x,$q3_y,0,0,$this->inpercent,$this->bandwidth_in,$q3_angle);
</code><br />
那么在draw_curve()前面,依葫芦画瓢来上一段就好了:<br />
<code class="highlighter-rouge">php
if(($this->bandwidth_out == '0') && ($this->bandwidth_in == '0'))
{
$link_width='0';
}
else
{
$link_width=$this->width;
}
draw_curve($im, $this->curvepoints, $link_width, $outline_colour, $comment_colour, array($link_in_colour, $link_out_colour), $this->name, $map);
</code></p>
<p>这个改完当然不能看到效果,看到就该排障去了~不过可以采用一点变通的方法来验证一下,比如某些线路现在比较空闲,我们就把预定的阀值0(断网的流量)改大一些,刚好超过某个空载线路即可了:</p>
<p>比如改成 <code class="highlighter-rouge">if ( $this->bandwidth_out < '1000' )</code> 的话,weathermap效果变成如下:</p>
<p><img src="/images/uploads/qqe688aae59bbee69caae591bde5908d.jpg" alt="wethermap" /></p>
<p>其中两条流量小于1000bits的线路,其width就变成了0,只留下一条直线了~~</p>
<p>至于为了 <code class="highlighter-rouge">width==0</code> 却还留下了一条线,这就跟weathermap的绘图方式有关了。<code class="highlighter-rouge">draw_curve</code> 中是这么利用gd画图的:</p>
<ol>
<li>根据node的位置,获取在二维空间上的x轴、y轴坐标,在两个node之间通过打点的方式进行矢量连线;</li>
<li>获取设定的width,将之前的坐标点平移相应的位置,再次打点;</li>
<li>在连线的中点处绘制箭头;</li>
<li>填充颜色。</li>
</ol>
<p>如果要完全去除掉这根连线,或许可以在 <code class="highlighter-rouge">draw_curve</code> 中设定其连线长度为0?在 <code class="highlighter-rouge">draw_curve()</code> 中有一个变量叫 <code class="highlighter-rouge">$totaldistance</code>,指的是两个node之间的距离,之后包括箭头、文字等,都是以这个变量*50%来计算的。添加 <code class="highlighter-rouge">if($this->bandwidth_out<1000){$totaldistance=0}</code>,等了十分钟再刷新,可图片依然没有更新!</p>
<p>继续看 <code class="highlighter-rouge">$totaldistance</code> 是怎么得出的,看到了 <code class="highlighter-rouge">$this->curvepoints</code>,而 <code class="highlighter-rouge">$this->curvepoints</code> 是通过 <code class="highlighter-rouge">calc_curve($xpoints, $ypoints)</code> 返回的。那么在 <code class="highlighter-rouge">Draw()</code> 中继续修改 <code class="highlighter-rouge">$this->curvepoints</code> 即可。变通一下上面的测试代码如下:<br />
<code class="highlighter-rouge">php
$link_width=$this->width;
$this->curvepoints = calc_curve($xpoints, $ypoints);
if ( $this->bandwidth_out < '2000' )
{
$link_width=0; //没有连线的话,宽度设啥都一样了~
$this->curvepoints = array( );
}
</code><br />
稍后刷新页面,看到原来的连线已经不见了~ 不过从效果图来看,少了连线反而不起眼了,还不如留着一根线容易引起警觉……</p>
<p><img src="/images/uploads/qqe688aae59bbee69caae591bde5908d1.jpg" alt="new-weathermap" /></p>
从猫扑论坛看终极页的缓存控制
2010-12-12T00:00:00+08:00
CDN
http://chenlinux.com/2010/12/12/learning-cache-control-from-mop-com
<p>GF找我要猫扑账号,只好去申请了一个,顺带着之前分析天涯的劲头,把猫扑也看看~</p>
<p>猫扑的左右分栏与天涯等论坛都不同,其左侧栏提供了大量推荐文章和栏目列表,右侧栏作为具体内容的阅读使用,通过firebug和源码阅读可以看到,左侧栏是通过frame加载的 <code class="highlighter-rouge">/leftFrame.jsp?type=*</code> ,再由该jsp根据后面的参数来调用相应的html页;右侧栏是通过js中定义的 <code class="highlighter-rouge">openRightUrl</code> 来打开具体某个html帖子,这个 <code class="highlighter-rouge">openRightUrl</code> 定义在<a href="http://txt.mop.com/dzhjs/dzh2js/turlUrl.js?version">http://txt.mop.com/dzhjs/dzh2js/turlUrl.js?version</a>里,其实就是一个 <code class="highlighter-rouge">right.location.href</code>。显然这个js将成为上猫扑时最常用的文件,其缓存时间相当长~header中可以看到 <code class="highlighter-rouge">Cache-Control: max-age=8640000</code></p>
<p>另,虽然header中的Server内容被修改,但当不带参数访问 <code class="highlighter-rouge">/leftFrame.jsp</code> 时,其调用的 <code class="highlighter-rouge">*_0_0.html</code> 不存在,显示出了 nginx0.7.34 的 404 错误页面。但随意输入 abcd.html,却能返回猫扑自定的错误页面,这也是比较怪的一点,怀疑其 nginx 的 proxy 配置不太正确。</p>
<p>最后还是看html的缓存控制和回复时的控制。</p>
<p>从列表中复制具体一个帖子的html链接地址另行打开,比如 <code class="highlighter-rouge">http://dzh.mop.com/topic/readSub_12990049_0_0.html</code> 第一个数应该是帖子号,第二个数是页码(从0开始计数~),第三个未知……</p>
<p>可以看到这该域名下的html的默认配置,max-age是180。</p>
<p>然后写下评论,点击提交回复。内容随即更新了,但url抓起到的url却是 <code class="highlighter-rouge">http://dzh.mop.com/topic/readSub_12990049_-1_0.html</code> ,而这个url的max-age设定是0!<br />
比较奇怪的是,回复时POST提交的url并没有和天涯一样返回一个302指向,而是返回一个无内容的200,但页码依然跳转了,而且 <code class="highlighter-rouge">-1_0.html</code> 的refer也不是POST的jsp,而是原先停留的 <code class="highlighter-rouge">0_0.html</code> ……</p>
<p>另外点了一下全文观看,其页码是-2,max-age也是180,显然回复后的显示url是特意定制的header~~</p>
<p>最后,有图有真相:</p>
<p><img src="/mop.jpg" alt="mop" /></p>
xen的dom0内存设置
2010-12-10T00:00:00+08:00
cloud
xen
http://chenlinux.com/2010/12/10/xen-dom0-memory-setup
<p>公司测试环境使用了xen来提供大批逻辑隔离的服务器供内部调测使用。随着应用系统和同事人数的增加,虚拟机数量越来越多,原先每台server开两三个vm已经吃紧,遂要增加新的vm。cp相应的img后,启动却失败了。</p>
<p>连上server看了一下具体的报错和系统信息,server的BIOS上识别出了1G内存,free命令的mem-total是491MB,xm li看到dom0的mem正是491M,而dom1、dom2各255M。</p>
<p>修改/etc/grub.conf,在kernel /xen.gz-2.6.18-8.el5后面加上dom0_mem=128M。reboot之后再启动第三台vm,OK!</p>
<p>另:按照一般经验,一台2G的server,dom0最低使用mem在128-256M,单个vm的mem最低为64M(debian)或128M(CentOS)</p>
从天涯论坛看终极页的缓存控制
2010-12-10T00:00:00+08:00
CDN
http://chenlinux.com/2010/12/10/learning-cache-control-from-tianya-cn
<p>一般不太上天涯论坛灌水或者潜水,不过经常去天涯SA刘天斯的blog上逛逛~在他开源memlink后,想起来去天涯看看前端设计,发现其论坛主列表页采用nginx发布(预计有nginx的module直接读取memlink),终极页前端采用varnish缓存,回复时的动态asp页面由IIS处理。但在终极页上,虽然显示的server也还是IIS,我却有一定的怀疑~</p>
<p>在访问某终极页时,可以看到类似如下的header<br />
Age:78<br />
Cache-Control:public<br />
Connection:close<br />
Content-Encoding:gzip<br />
Content-Length:34687<br />
Content-Type:text/html<br />
Date:Fri, 10 Dec 2010 09:03:48 GMT<br />
Expires:Fri, 10 Dec 2010 09:07:30 GMT<br />
Last-Modified:Fri, 10 Dec 2010 09:02:30 GMT<br />
Server:Microsoft-IIS/6.0<br />
Vary:Accept-Encoding<br />
Via:Tianya Cache<br />
X-Cache:HIT118<br />
X-Powered-By:ASP.NET<br />
X-tianya:1098678484 1098675384</p>
<p>如果按下F5,会看到一个IMS请求,最后返回304或者200的结果。</p>
<p>如果按下Ctrl+F5,会看到一个no-cache请求,最后返回200的结果。</p>
<p>不过奇怪的是,在no-cache请求后,虽然明知页面没有变(请求的是一个多页帖子的第一页,wget和wget –header ‘Cache-Control: no-cache’下来后的页面的MD5值都一样),但返回的last-modified时间却变成了和Date一致的当前时间了。以至于让我怀疑这个*.shtml难道不是静态化生成的?</p>
<p>然后回复该帖。通过POST方式向另一个动态域名传输数据,并302跳转回原页面。由于POST方式的不可缓存性,浏览器自动带上了no-cache请求头,并传递给了302之后的动作,即以no-cache重新请求了原帖子的url,并重新下载了该页面。由此完成了对回复的即时更新——对于论坛来说,重要的就是发帖人自己能即时看到,其他人完全可以等一会页面过期或者IMS比对来看别人的新回复。</p>
<p>假设前面说到的shtml确实是静态化生成的话,那么这个直接跳转的做法就有一定的风险,即要求系统在极短时间(从浏览器时间看就是POST的firstbyte时间开始,到200的connection时间结束,网络较好的情况下应该是毫秒级)内,完成对终极页的静态化工作。<br />
写到这里,愈发怀疑这个shtml是asp的伪装版了……</p>
flash绘图利器-amcharts
2010-12-10T00:00:00+08:00
web
amcharts
http://chenlinux.com/2010/12/10/intro-amcharts
<p>作为SA最常用的绘图工具肯定是rrdtool;而coder最常用的肯定是gd;前段时间从ibm文库里学到一个同样很强大的函数绘图工具gnuplot;最近在技术群里又见识到一些更新奇小巧的绘图工具,记录一下~</p>
<p>google的API提供一个简易而不失美观的方法(详见<a href="http://code.google.com/intl/zh-CN/apis/charttools/docs/choosing.html">http://code.google.com/intl/zh-CN/apis/charttools/docs/choosing.html</a>)只需要在url的strings里提供一些数据,google就能直接返回给你想要的图表。当然也能采用google提供的js库完成高级一些的内容——不过对某些小心翼翼的SA来说,采用外部js总让人有一种不安全感~而比较固定的样式又可能不足以让销售部门的同事满意……</p>
<p>扶凯大大及时冒泡,提供给大家另一个工具:amcharts!一个flash绘图工具。flash的美观度和互动能力绝对是众所周知的有保证~(详见<a href="http://www.amcharts.com/">http://www.amcharts.com/</a>)官网界面简洁明快,各种examples,包括股价图、柱状图、饼状图、线面图、散点图,网状图,支持3D效果、渐变效果、背景图自定义、指针自定义图文提示、数据动态输入等各种功能。xml配置文件的每个标签都有详细注释。使用时,只需要在webroot下放上amcharts的swf,写好settings.xml,在html里插入swf即可~</p>
<p>在选择到stock的Smoothed line chart(平滑线图)时,我很惊讶的发现:这不就是之前我一直很赞叹的蓝汛客户服务平台里的带宽flash图么?如下所示:</p>
<p><img src="/images/uploads/charts.jpg" alt="amcharts" /></p>
<p>不过这个工具虽然千好万好,没想出来对我目前的工作有啥使用的必要……姑且记录之~</p>
nagios的add-ons安装小抄~
2010-12-04T00:00:00+08:00
monitor
nagios
http://chenlinux.com/2010/12/04/nagios-add-ons-install
<p>给nagios安装几个add-ons,碰到一些一般安装教程上不会提及的问题,记录一下:</p>
<p>1、ndoutils:<br />
在./configure通过后make一直error,因为不管是否–with-mysql-lib,也不管–with-mysql-lib=/usr/local/mysql/lib还是/usr/local/mysql/lib/mysql,甚至使用LDFLAGS=-I/usr/local/mysql/lib等等,最后在make的时候总还是会报出如下错误:<br />
../include/config.h:261:25: error: mysql/mysql.h: No such file or directory<br />
../include/config.h:262:26: error: mysql/errmsg.h: No such file or directory<br />
解决办法:编辑config.h文件的261和262行,把mysql/*.h的mysql/删除掉即可。</p>
<p>make完成后,将相应文件cp到指定目录,启动ndomod会报错libmysqlclient.so.6.0.0动态链接库无法找到。网上一般都说ln -s /usr/local/mysql/lib/* /usr/lib;echo ‘/usr/lib’ » /etc/ld.so.conf;ldconfig即可。其实还不行。<br />
解决办法:echo ‘/usr/local/mysql/lib/mysql’ > /etc/ld.so.conf.d/mysql.conf;ldconfig即可。因为通用办法的目录不够深。</p>
<p>2、pnp4nagios:<br />
之前使用的pnp0.4.<em>版本,今天下的是pnp0.6.</em>版本。整个url设计发生了较大变化。各监控项页面的url从pnp4nagios/index.php?host=&service=变成了/pnp4nagios/graph?host=&service=。而这个graph(rrd图像的url是/pnp4nagios/image?***)则通过apache的mod_rewrite实现。<br />
pnp自己编译时可以make出来一个httpd.conf。其中相关Rewrite的包括:<br />
<code class="highlighter-rouge">apache
<Directory '"/usr/local/pnp4nagios/share/">
RewriteEngine On
RewriteBase /pnp4nagios/
RewriteRule ^(application|modules|system) - [F,L]
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule .* index.php$0 [PT,L]
</Directory>
</code><br />
因为pnp编译时没有具体区分etc和share的路径,所以之后apache的发布路径也不同,为了方便,不再写directory。最后经过反复试验,可用配置如下:<br />
<code class="highlighter-rouge">apache
RewriteEngine On
RewriteRule ^(application|modules|system) - [F,L]
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-d
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule ^/nagios/pnp4nagios/(.*) /nagios/pnp4nagios/index.php$1 [PT,L]
</code></p>
<p>试验中发现几个apache与nginx的不同:<br />
1、rewriterule的转向url不能用^标记起始端!<br />
2、PT的强制进入下一个处理器相当有用,不然会形成回环rewrite!<br />
3、rewritecond里德%{REQUEST_FILENAME}默认是在rewritebase下的——而rewritebase只能在directory里使用。</p>
keepalived故障一例
2010-12-03T00:00:00+08:00
linux
http://chenlinux.com/2010/12/03/keepalived-problem-about-vrrp-delay
<p>一组lvs,以keepalived主从方式运行。今天早上突然收到VIP报警,所有的VIP都ping不通了。</p>
<p>上lvs运行ipvsadm -ln一看,计数表全是0!怀疑是slave霸占了MAC,于是keepalived restart了一次,故障依旧。然后再restart了一次master,顿时恢复正常。</p>
<p>然后分析messages中的详细信息,推理本次故障的过程如下:</p>
<p>故障前——masterA,slaveB</p>
<div class="highlighter-rouge"><pre class="highlight"><code>0:00 A "kernel:eth1:link DOWN",疑似网卡物理中断,自动降级成slave;
0:01 B检测A宕机,升级为master,send arp到交换机,add所有VIP;
0:40 A "kernel:eth1:link UP",但配置应该是不自动抢占;
0:43 A检测B宕机,升级为master;
0:48 A发送arp刷新请求到交换机,add所有VIP,但因为是物理中断,目前A实际仍处于断网状态,有期间RIP检查的timeout为证;
1:10 A检测RIP的http status正常,即此时A的网络才正式恢复正常;
1:10 B检测发觉A的状态为master,降级为slave,remove所有VIP;
</code></pre>
</div>
<p>在0:43的时候,masterA的ARP刷新请求没能发送到交换机,而交换机记录的对应地址就还是B的——但在1:10时,B自认为slave而移除了所有ip。导致ping失败!</p>
<p>故障解决过程解析:</p>
<p>1、重启B——因为B已经是slave,所以restart不会发送ARP刷新,无效; <br />
2、重启A——因为A自认是master,重启A导致keepalived切换,会触发B发送ARP刷新,恢复正常。</p>
<p>最终解决办法:</p>
<p>在keepalived.conf中添加garp_master_delay 30参数,让slave在升级成master后延时30s再发送一次arp刷新请求,以应对网卡硬件中断引起的这个问题。</p>
weathermap-cacti-plugin学习(2)
2010-11-26T00:00:00+08:00
monitor
cacti
php
http://chenlinux.com/2010/11/26/learning-weathermap-cacti-plugin-2
<p>在Weathermap.class.php中,定义了一个function叫LoadPlugins,读取lib/datasources/下的php类。其中就有WeatherMapDataSource_rrd.php。其中定义了Init、Recognise和ReadData三个方法。明显是ReadData函数来读取rra数据,具体方法为调用管道,运行rrdtool命令。</p>
<p>命令如右:rrdtool fetch *.rrd AVERAGE –start now-800 –end now</p>
<p>然后是对命令的结果进行分析。可以先运行一下看看效果:</p>
<p>[raochenlin@cacti datasources]$ date +%s;/usr/local/rrdtool-1.2.18/bin/rrdtool fetch /www/cacti/rra/10_168_168_130_traffic_in_2866.rrd AVERAGE –start now-800 –end now<br />
1290704787<br />
traffic_out traffic_in</p>
<p>1290704100: 4.8623533333e+01 2.2821386667e+02<br />
1290704400: 4.0663000000e+01 1.9051346667e+02<br />
1290704700: 3.5332000000e+01 2.3380053333e+02<br />
1290705000: nan nan</p>
<p>由此可见数据是300s一采集,所以当设定start是-800的时候,就会取比800大的离800最近的300的倍数,即900之间的数据。</p>
inotify-purge后续分析
2010-11-25T00:00:00+08:00
linux
inotify
bash
gnuplot
http://chenlinux.com/2010/11/25/draw-images-of-inotify-log-by-gnuplot
<p>在选定sersync2进行command方式刷新后,需要对诸多域名的更新频率做个简单的分析,以了解编辑的操作习惯,方便选定调整时间、确定文件过期时间等等。</p>
<p>经过测试,早command插件下,sersync输出的是file的全路径。考虑到实际情况是有大量servername和serveralias,运行如下脚本:<br />
```bash<br />
#!/bin/bash<br />
MODIFY_FILE=”$1”<br />
MODIFY_DIR=<code class="highlighter-rouge">echo $MODIFY_FILE|awk -F/ '{print $3}'</code><br />
MODIFY_URI=<code class="highlighter-rouge">echo $MODIFY_FILE|sed 's/\/backup\/.*.test.com\/htdocs//'</code><br />
MODIFY_DOMAIN=<code class="highlighter-rouge">cat servername.list|grep $MODIFY_DIR|awk '{print $2}'</code></p>
<p>IPLIST=”1.1.1.12<br />
1.1.1.79<br />
1.1.1.87<br />
1.1.1.21<br />
1.1.1.22<br />
1.1.1.23<br />
1.1.1.27<br />
1.1.1.80<br />
“<br />
Time=<code class="highlighter-rouge">date +%Y%m%d</code><br />
Username=’test.com’<br />
Userkey=’test’<br />
Userpass=’test.com1234’<br />
MD5=<code class="highlighter-rouge">echo -n "$Time$Username$Userkey$Userpass"|md5sum|awk '{print $1}'</code></p>
<p>function cache_purge {<br />
for i in $IPLIST;do<br />
/home/tools/squidclient -p 80 -h $i -m purge “$1”<br />
done<br />
curl -s -G -d “username=${Username}&md5=${MD5}&url_list=$1” http://cs.fastweb.com.cn/interface/push_portal.php<br />
}</p>
<p>for i in <code class="highlighter-rouge">echo $MODIFY_DOMAIN|sed 's/,/ /g'</code>;do<br />
PURGE_URL=”http://$i$MODIFY_URI”<br />
echo “<code class="highlighter-rouge">date +%F-%T</code> $PURGE_URL” » purge.log<br />
cache_purge $PURGE_URL<br />
done<br />
```</p>
<p>然后运行如下命令,分别得到html/js/css的每分钟更新量:(有点小瑕疵,即当某分钟html无更新时js和css也无法记录,不过这种概率应该不高)</p>
<div class="highlighter-rouge"><pre class="highlight"><code>cat /home/tools/purge.log |awk -F<span class="s2">"[:|-]"</span> <span class="s1">'/html/{a[$4":"$5]++}/js/{b[$4":"$5]++}/css/{c[$4":"$5]++}END{for(i in a){print i,a[i],b[i],c[i]}}'</span>|sort
</code></pre>
</div>
<p>得到文件类似如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>11:23 28 15
11:24 10 7
11:25 224 37 13
11:26 470 192
11:27 344 187 1
11:28 441 77 2
11:29 419 8
</code></pre>
</div>
<p>然后创建gnuplot.conf如下:<br />
<code class="highlighter-rouge">tcl
set terminal png xFFEEDD size 2048,512
set output "log.png"
set autoscale
set xdata time
set timefmt "%H:%M"
set format x "%H:%M"
set xtics 10
set mxtics 4
set style data lines
set datafile missing "0"
set xlabel "time per day"
set ylabel "purge"
set title "DPD expires"
set grid
plot "log" using 1:2 title "html/min","log" using 1:3 title "js/min","log" using 1:4 title "css/min"
</code></p>
<p>运行 <code class="highlighter-rouge">cat gnuplot.conf|gnuplot</code> 就得到 log.png 了,如下:</p>
<p><img src="/images/uploads/log.png" alt="gnuplot-log" /></p>
weathermap-cacti-plugin学习(1)
2010-11-24T00:00:00+08:00
monitor
cacti
php
http://chenlinux.com/2010/11/24/learning-weathermap-cacti-plugin-1
<p>weathermap是一个利用php的gd库画图的程序,它可以自主运行,但更多情况下是作为cacti等监控工具的插件,通过rra数据库获取数据完成绘图。其官网地址如右:<a href="http://www.network-weathermap.com">http://www.network-weathermap.com</a></p>
<p>正常情况下,其link的color是由浅到深,当链路接近满载时,显示为鲜红色~然而这个设定遗漏了一个更加严重的情况——链路中断!在完全没有流量的情况下,weathermap应该会按照默认的0%处理——显示为白色(有+1的黑色边框)。</p>
<p>当然,这么可怕的事情,肯定会有多种手段来完成报警,不至于靠人眼盯着weathermap来汇报。但毕竟算是个功能上的缺失。</p>
<p>很巧,在zenoss(和cacti类似的另一款监控软件)的wiki上,看到有网友修改的perl版的weathermap,网址如右:<a href="http://community.zenoss.org/docs/DOC-2543">http://community.zenoss.org/docs/DOC-2543</a>。其配置文件中的WIDTH标签,比php的多出了<%status-width(device name,component name)%>配置,其解释说“draw link with width 0 if it is down otherwise draw it with width_ok width”。相关代码如下:<br />
<code class="highlighter-rouge">perl
while($line=~m/<%status-width([^%]+)%>/)
{
my $tmp;
$tmp=$1;
my $res;
#WidthIfOK, device, port
if ($tmp=~m/\((.+),(.+),(.+)\)/)
{
my $url=get_device_url($2);
$res=get_port_status("$url/$3");
if ($res==1) #OK
{
$res=$1; #default width
}
else
{
$res=0;
}
}
else
{
die ("Bad format $line");
}
if ($line!~s/<%status-width([^%]+)%>/$res/)
{
die ("Error 1");
}
}
</code><br />
思路是通过wget数据获取状态,一旦错误就至width为0,否则读取正常设定值绘图。</p>
<p>在原版的php中相关部分如下:<br />
<code class="highlighter-rouge">php
if (preg_match("/^\s*WIDTH\s+(\d+)\s*$/i", $buffer, $matches))
{
if ($last_seen == 'LINK')
{
$curlink->width=$matches[1];
$linematched++;
}
else // we're talking about the global WIDTH
{
$this->width=$matches[1];
$linematched++;
}
}
</code><br />
显然只要在这里加上一个else{}就可以了。</p>
<p>至于如何判定链路中断,有待继续学习~是外挂一个ping?或者读取rra中的数值?下一步先看懂Weathermap.class.php是怎么读取rra数值的吧~</p>
<p>(题外话,在baidu该字眼的第一页结果,看到中南民族大学的校园网cacti页面。他们居然开放匿名访问,甚至settings都能点开,无语~网址如右:<a href="http://210.42.159.3/cacti/plugins/weathermap/weathermap-cacti-plugin.php">http://210.42.159.3/cacti/plugins/weathermap/weathermap-cacti-plugin.php</a>)</p>
ftp中的软连接问题
2010-11-22T00:00:00+08:00
linux
http://chenlinux.com/2010/11/22/pureftpd-problem-about-symbolic-links
<p>数据临时迁移,为了尽量不影响业务,创建了一个软连接。不料pureftpd出了一点小问题。</p>
<p>当ln -s /text /www/text的时候,如果pureftpd.passwd中指定的是/www/text,访问没有问题;如果是/www的话,再cd text会出问题。</p>
<p>搞怪的是,text1出的报错是Too many levels of symbolic links,而text2出的报错是No such file or directory……</p>
<p>进pureftpd的src里看./configure –help,看到如下一行:</p>
<p>–with-virtualchroot Enable the ability to follow symlinks outside a chroot jail</p>
<p>以前的编译,都只用了–with-everything Build a big server with almost everything</p>
<p>看来almost里还真就不包括virtualchroot……</p>
<p>重新编译pureftpd,加上了virtualchroot参数。然后cp原ftp的pdb过来,启动一试,OK~</p>
sersync2.5试用~
2010-11-19T00:00:00+08:00
linux
inotify
http://chenlinux.com/2010/11/19/intro-sersync2-5
<p>之前采用 inotify-tools 来触发 purge,实际运行中碰上一些问题:</p>
<p>1、因为一个文件的更新,可能伴随着 <code class="highlighter-rouge">create</code>、<code class="highlighter-rouge">modify</code>、<code class="highlighter-rouge">close_write</code> 等等过程,而文件稍大一些,甚至就有连续的 <code class="highlighter-rouge">modify</code> 出现。于是一个文件的更新,通过管道发送的 purge 请求经常可以看到三四个!如果量不大的情况下,倒也没什么,可如果更新比较频繁的情况下,再翻上三四倍,任务就比较拥挤了。</p>
<p>2、cms 程序通常是通过 ftp 等方式,将页面上传的文件 move 到指定位置。而 ftp 本身在接受过程中也会产生名叫 <code class="highlighter-rouge">.pureftpd-upload.****</code> 的临时文件。</p>
<p>第二个倒可以在管道后面再 grep 一次解决,但第一个问题在管道这种流形式的简单处理中就没办法了。预想了几个办法,先一个是记录日志,然后每次管道接受时比对日志中最近十条是否有重复,不过这么频繁的读写日志文件,也会很郁闷~;后一个是把日志文件转进内存去,即管道获取的信息 push 进一个 hash,然后 sleep 后再 poll 这个 hash 出来,不过 shell 没有 hash,就得把简单的 shell 重写成比较复杂的 perl 了……</p>
<p>今天刚想起来半年前曾经看到过金山逍遥运维部开源的一个项目,也是基于 inotify 完成的。去翻来看看,很好很强大,感觉完全能满足我目前的想法。</p>
<p>项目网址:<a href="http://code.google.com/p/sersync/">http://code.google.com/p/sersync/</a></p>
<p>提供了 bin 和 src 两个版本,直接下 bin 来用:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget http://sersync.googlecode.com/files/sersync2.5_64bit_binary_stable_final.tar.gz
tar zxvf sersync2.5_64bit_binary_stable_final.tar.gz
<span class="nb">cd </span>GNU-Linux-x86/
</code></pre>
</div>
<p>很简单,只有两个文件,一个是程序,一个是 xml 配置文件。配置包括 debug 模式、xfs 支持、过滤器配置(默认已过滤<code class="highlighter-rouge">^.</code>和<code class="highlighter-rouge">~$</code>)、inotify 监听(推荐是创建目录、完成输入和移动)、本地监听路径和 rsync 远程主机(ip,rsync 模块名、用户名密码)、失败重试及日志、多次失败后的定时任务、插件(通过socket 向远程主机传输 inotify 日志、通过 http 向 cdn 的 api 发送 purge 请求、调用外部命令处理文件)。</p>
<p>本来直接就有 purge 功能,可惜我的环境下域名比较多,目前的功能上只能对 url 做 regex,不能反引用到 domain 上。所以是写个 shell,然后采用 command 插件传递参数~</p>
<p>先最简单的实验,写个write.sh如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#!/bin/sh</span>
<span class="nb">echo</span> <span class="nv">$1</span> >> <span class="nv">$0</span>.log
</code></pre>
</div>
<p>然后修改xml如下句:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="nt"><param</span> <span class="na">prefix=</span><span class="s">"GNU-Linux-x86/write.sh"</span> <span class="na">suffix=</span><span class="s">""</span> <span class="na">ignoreError=</span><span class="s">"true"</span><span class="nt">/></span>
</code></pre>
</div>
<p>运行 <code class="highlighter-rouge">GNU-Linux-x86/sersync2 -d -m command</code> 即可后台运行 command 插件且不启用 rsync。</p>
<p>然后 <code class="highlighter-rouge">tailf write.sh.log</code> 看,果然每条 url 都不重复了~~</p>
<p>(看了作者周洋的 blog,其中提到文件如果比较大,更新完成时间超过一定值,也会导致队列重复,我猜估计思路和我的第二种想法应该是类似的。)</p>
<p>另,<code class="highlighter-rouge">sersync -h</code> 可以看到其固定修改 sysctl 如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nb">echo </span>50000000 > /proc/sys/fs/inotify/max_user_watches
<span class="nb">echo </span>327679 > /proc/sys/fs/inotify/max_queued_events
</code></pre>
</div>
<p>据周洋的说法是 inotify 最多只能监听到五千万个文件夹~~在我的环境下,1300 万 inode,add watch 就花了1个多小时……</p>
pnp4nagios的模板问题(2)
2010-11-18T00:00:00+08:00
monitor
nagios
rrdtools
php
http://chenlinux.com/2010/11/18/template-problem-of-pnp4nagios-2
<p>接上篇。结尾时找到的url确实就是解决这个问题的。我错怪作者鸟~~</p>
<p>在 <code class="highlighter-rouge">nagios/etc/pnp/check_commands</code> 文件夹下,可以添加 <code class="highlighter-rouge">%check_command%.cfg</code> 配置文件,以定义pnp在使用模板时的一些参数。在我的应用环境下,只需要添加如下一个 <code class="highlighter-rouge">check_nrpe.cfg</code> 即可:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>CUSTOM_TEMPLATE = 1 #使用命令的第一个参数做自定义模板名
DATATYPE = GAUGE #数据类型为即时数值
USE_MIN_ON_CREATE = 0 #绘图数据最小值为0,用来排除某些错误溢出导致的负值
</code></pre>
</div>
<p>然后就可以进 <code class="highlighter-rouge">nagios/share/pnp/templates/</code> 下去创建自己需要的模板了。仿照cacti的样子写个loadavg的如下:<br />
<code class="highlighter-rouge">php
<?php
$opt[1] = "--vertical-label Load -l0 --title \"CPU Load for $hostname / $servicedesc\" ";
$def[1] = "DEF:var1=$rrdfile:$DS[1]:AVERAGE " ;
$def[1] .= "DEF:var2=$rrdfile:$DS[2]:AVERAGE " ;
$def[1] .= "DEF:var3=$rrdfile:$DS[3]:AVERAGE " ;
$def[1] .= "CDEF:total=var1,var2,+,var3,+ " ;
$def[1] .= "HRULE:$WARN[1]#FFFF00 ";
$def[1] .= "HRULE:$CRIT[1]#FF0000 ";
$def[1] .= "AREA:var1#FFD700:\"load 1 \" " ;
$def[1] .= "GPRINT:var1:LAST:\"%6.2lf last\" " ;
$def[1] .= "GPRINT:var1:AVERAGE:\"%6.2lf avg\" " ;
$def[1] .= "GPRINT:var1:MAX:\"%6.2lf max\\n\" ";
$def[1] .= "AREA:var2#7FFF00:\"Load 5 \":STACK " ;
$def[1] .= "GPRINT:var2:LAST:\"%6.2lf last\" " ;
$def[1] .= "GPRINT:var2:AVERAGE:\"%6.2lf avg\" " ;
$def[1] .= "GPRINT:var2:MAX:\"%6.2lf max\\n\" " ;
$def[1] .= "AREA:var3#FF0000:\"Load 15\":STACK " ;
$def[1] .= "GPRINT:var3:LAST:\"%6.2lf last\" " ;
$def[1] .= "GPRINT:var3:AVERAGE:\"%6.2lf avg\" " ;
$def[1] .= "GPRINT:var3:MAX:\"%6.2lf max\\n\" " ;
$def[1] .= "LINE1:total#000000 " ;
?>
</code></p>
<p>采用DEF方式定义数据,CDEF方式计算总和,HRULE画水平线,AREA画涂层,LINE画连线(有1/2/3种粗细),STACK累加数值绘图效果,GPRINT计算并显示数据。</p>
<p>只画一个图就都是$def[1],如果需要两个图就是$def[2],依次类推,不过页面上的Datesource是读取的*.xml文件中的<name>,如果合并数据绘图,显示就会有问题,所以最好就把所有数据都画一张图里。比如网卡流量。用CDEF取反,我不知道RPN中有没有特定函数,简单的采用了0,var,-方式,将部分数据倒到x轴下方~~</name></p>
<p>如果不是STACK的话,需要注意一点,AREA方式是不透明的,单纯的图层覆盖,所以一定要把最大的值放在最前面绘制,然后才能有效果。像流量这种没谱的事情,最好就采用LINE方式~~</p>
<p>目前我绘制的load、conn、flow三个rra如下:</p>
<p><img src="/images/uploads/load.jpg" alt="load" /></p>
<p><img src="/images/uploads/conn.jpg" alt="conn" /></p>
<p><img src="/images/uploads/flow.jpg" alt="flow" /></p>
pnp4nagios的模板问题(1)
2010-11-17T00:00:00+08:00
monitor
nagios
rrdtools
php
http://chenlinux.com/2010/11/17/template-problem-of-pnp4nagios-1
<p>实在是看不下去pnp4nagios的丑陋的rrd效果,想着自己修改一下模板。</p>
<p>很容易找到了目前使用的模板:nagios/share/pnp/templates.dist/default.php,不过看到目录下已经有不少其他的模板文件,为啥一个都没用上呢?</p>
<p>从nagios/share/pnp/include/function.inc.php里看到了doFindTemplate(),定义如下:<br />
<code class="highlighter-rouge">php
if (is_readable($conf['template_dir'].'/templates/' . $template . '.php')) {
$template_file = $conf['template_dir'].'/templates/' . $template . '.php';
}elseif (is_readable($conf['template_dir'].'/templates.dist/' . $template . '.php')) {
$template_file = $conf['template_dir'].'/templates.dist/' . $template . '.php';
}elseif (is_readable($conf['template_dir'].'/templates/default.php')) {
$template_file = $conf['template_dir'].'/templates/default.php';
}else {
$template_file = $conf['template_dir'].'/templates.dist/default.php';
}
</code><br />
也就是说其实pnp是找不到对应check_command的模板,才使用了最后的default.php!</p>
<p>正巧,default.php里输出了check_command到rra上,可以看到,几乎所有的命令输出都是”Check Command check_nrpe”。</p>
<p>也就是说,pnp设计者很贴心的设计了自动查找命令模板的功能,却没有考虑到nagios最广泛的应用插件nrpe……</p>
<p>在pnp官网文档<a href="http://docs.pnp4nagios.org/pnp-0.4/tpl?s[]=template">http://docs.pnp4nagios.org/pnp-0.4/tpl?s[]=template</a>上指出,rrd使用模板是读取了perfdata中相应的xml文件,xml内容类似如下几行:<br />
```xml<br />
<?xml version="1.0" encoding="UTF-8" standalone="yes"?></p>
<nagios>
<datasource>
<template>check_nrpe</template>
<is_multi>0</is_multi>
<ds>1</ds>
<name>eth0_in</name>
<unit>Mbps</unit>
<act>2.96</act>
<warn>976.56</warn>
<warn_min></warn_min>
<warn_max></warn_max>
<warn_range_type></warn_range_type>
<crit>976.56</crit>
<crit_min></crit_min>
<crit_max></crit_max>
<crit_range_type></crit_range_type>
<min>0</min>
<max>0</max>
</datasource>
```
这些标签都是给模板使用的,比如default.php中输出命令的那行就是:
$def[$i] .= 'COMMENT:"Check Command ' . $TEMPLATE[$i] . '\r" ';
如果直接修改xml中的<template>标签内容,确实可以调用成新的模板显示。但间隔时间一过,xml就自动更新成默认配置输出的结果……
解决在大规模环境下nrpe监控数据绘图模板的问题~或许还得继续查找xml的定义,待续ing~
补充:看到官网如下网页,似乎是针对这个问题的,英文慢慢啃~
<a href="http://docs.pnp4nagios.org/pnp-0.4/tpl_custom">http://docs.pnp4nagios.org/pnp-0.4/tpl_custom</a>
</template></nagios>
基调发布《中国互联网网站性能行业参考数据》
2010-11-16T00:00:00+08:00
CDN
http://chenlinux.com/2010/11/16/china-internet-sites-performance-industry-reference-data-publiced-by-networkbech
<p><a href="http://www.networkbench.com/trade-rank/index.html">http://www.networkbench.com/trade-rank/index.html</a></p>
<p>不是做广告,不过说实话还是觉得基调很有头脑,搞出这么个东西来~~~</p>
mysql状态报表工具
2010-11-12T00:00:00+08:00
database
MySQL
http://chenlinux.com/2010/11/12/intro-mysqlreport
<p>最近学习mysql,从监控的角度出发,然后发现了一个很不错的个人网站<a href="http://hackmysql.com">http://hackmysql.com</a>,啧啧,看这名字就NB烘烘滴~</p>
<p>网站的tools列表提供了一系列站长认为很不错的mysql工具,其中有他自己早年写的四个perl,也有他目前所在公司出品的工具(大名鼎鼎的Maatkit,据说80%的国外mysqlDBA使用它,国内有大头刚曾经写过14篇介绍文章,以后有时间再看,地址如下:<a href="http://search.chinaunix.net/bbs.php?q=Maatkit&username=&st=title&bbs=1&forums=136&page=1">http://search.chinaunix.net/bbs.php?q=Maatkit&username=&st=title&bbs=1&forums=136&page=1</a>)。</p>
<p>在国内目前的技术文章来看(即百度可见范围内),比较常见的两个工具正是该站提供的,一个是状态报告工具mysqlreport,一个是日志分析工具mysqlsla。</p>
<p>今天先说mysqlreport,安装很简单:wget <a href="http://hackmysql.com/scripts/mysqlreport">http://hackmysql.com/scripts/mysqlreport</a></p>
<p>要使用它,首先需要有几个perl模块:DBI和DBD::mysql。CPAN安装即可。需要注意的是,因为DBD::mysql的安装过程中需要调用mysql_config,如果机器上没有mysql或者mysql的bin不在PATH里,都会报错。这时候退出安装mysql,然后到.cpan/里去手动perl Makefile.PL –with-mysql_config=/path/to/mysql_config安装吧~~</p>
<p>使用方法也有–help的详尽说明,大抵是–host/–user/–password,比较好玩的是还提供了–relative/–report-count用来短时间段内的定时报告。</p>
<p>报告包括:</p>
<ul>
<li>key buffer的使用率和命中率;</li>
<li>请求的分类比例(QC Hits和DMS越多越好,Com_最好不要超过3%)及具体情况;</li>
<li>慢查询情况(最好一个没有);</li>
<li>全表查询和排序情况(这个越少越好);</li>
<li>表锁等待情况(最好没有);</li>
<li>表使用和命中率情况(尽量命中的好);</li>
<li>连接情况(适中即可);</li>
<li>线程复用情况;</li>
</ul>
<p>然后是InnoDB引擎的一些</p>
<ul>
<li>锁等待;</li>
<li>读写速度;</li>
<li>行操作情况</li>
</ul>
<p>mysqlreport2008年之后就停止了更新,一部分人则开始采用tuning-primer.sh收集报表。这个shell脚本直接利用mysql客户端登陆服务器后show status然后进行运算,除了和mysqlreport极为类似的报表外,还采用不同颜色显示提供了作者的优化建议,边看边学习,很不错,下载地址如下:<a href="http://www.day32.com/MySQL/tuning-primer.sh">http://www.day32.com/MySQL/tuning-primer.sh</a></p>
<p>这个网站同样提供了对mysql主从同步的脚本和监控脚本,都是shell脚本~~</p>
<p>最后,还要表扬一下ifeng的运维童鞋,他们用php完成一个功能介乎mysqlreport和tuning-primer之间的mysql状态报表网页,目前版本是mysqlmonitor1.0.0(不过下载链接坏了……)。其中很多推荐配置中文说明,不过在“具体情况具体分析”方面还不够智能,所有建议都是统一在4G内存mysql服务器的假设条件下给出的,也没有对报表数据进行阀值触发。</p>
nrpe编译小问题
2010-11-10T00:00:00+08:00
monitor
nagios
http://chenlinux.com/2010/11/10/problem-of-nrpe-compiling
<p>一般情况下,在nagios被监控端安装nrpe和nagios-plugins的工作相当的简单重复。不过这次碰上一个诡异问题。</p>
<p>设备是RedHat AS4,在./configure时,报出如下错误:</p>
<p>checking for C compiler default output file name… a.out</p>
<p>checking whether the C compiler works… configure: error: cannot run C compiled programs.</p>
<p>If you meant to cross compile, use `–host’.</p>
<p>See `config.log’ for more details.</p>
<p>dmesg信息输出如下:</p>
<p>a.out[4272]: segfault at 00000000bffff770 rip 0000000000400456 rsp 00000000bffff770 error 4</p>
<p>起先以为是内存问题,检查boot.log没问题;然后又yum reinstall了gcc,问题依旧。</p>
<p>在config.log中慢慢翻,赫然看到如下一段:</p>
<p>configure:1782: checking for C compiler version</p>
<p>configure:1785: gcc –version </dev/null >&5</p>
<p>gcc32 (GCC) 3.2.3 20030502 (Red Hat Linux 3.2.3-47.3)</p>
<p>Copyright (C) 2002 Free Software Foundation, Inc.</p>
<p>This is free software; see the source for copying conditions. There is NO</p>
<p>warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.</p>
<p> </p>
<p>configure:1788: $? = 0</p>
<p>configure:1790: gcc -v </dev/null >&5</p>
<p>Reading specs from /usr/lib/gcc-lib/x86_64-redhat-linux/3.2.3/specs</p>
<p>Configured with: ../configure –prefix=/usr –mandir=/usr/share/man –infodir=/usr/share/info –enable-shared –enable-threads=posix –disable-checking –with-system-zlib –enable-__cxa_atexit –enable-languages=c,c++ –disable-libgcj –host=x86_64-redhat-linux</p>
<p>Thread model: posix</p>
<p>gcc version 3.2.3 20030502 (Red Hat Linux 3.2.3-47.3)</p>
<p>configure:1793: $? = 0</p>
<p>configure:1795: gcc -V </dev/null >&5</p>
<p>gcc32: argument to `-V’ is missing</p>
<p>configure:1798: $? = 1</p>
<p>……</p>
<h2 id="section">—————–</h2>
<h2 id="output-variables">Output variables.</h2>
<h2 id="section-1">—————–</h2>
<p>……</p>
<p>host=’x86_64-unknown-linux-gnu’</p>
<p>真相大白!因为./configure检查出来的hostname和gcc编译时的hostname不一致!</p>
<p>改用./configure –host=x86_64-redhat-linux,编译顺利通过~~~</p>
squid的debug日志
2010-11-09T00:00:00+08:00
squid
http://chenlinux.com/2010/11/09/intro-squid-debug_log
<p>今天为了检查防盗链配置,打开了squid的debug日志(squid.conf: debug_options ALL,9)。tailf观察,很是学习了一番squid的工作模式。</p>
<p>1、空闲时日志也一直在滚动,诸如“do_comm_select: 0 fds ready”、“storeDirClean: Cleaning directory /tmpfs/cache/03/2D”这样,第一个很显然就是表示TCPsocket可以accept请求;第二个是定期清除过期目录;</p>
<p>2、发送请求之后,accept部分如下:</p>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>fd_open FD 15 HTTP Request</td>
<td> </td>
</tr>
<tr>
<td>2010/11/09 19:18:18</td>
<td>httpAccept: FD 15: accepted port 80 client 124.238.250.28:39168</td>
<td> </td>
</tr>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataLock: 0x7589c8</td>
<td> </td>
</tr>
<tr>
<td>2010/11/09 19:18:18</td>
<td>comm_add_close_handler: FD 15, handler=0x4218f0, data=0x9a14f8</td>
<td> </td>
</tr>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataLock: 0x9a14f8</td>
<td> </td>
</tr>
<tr>
<td>2010/11/09 19:18:18</td>
<td>commSetTimeout: FD 15 timeout 300</td>
<td> </td>
</tr>
<tr>
<td>2010/11/09 19:18:18</td>
<td>commSetSelect: FD 15 type 1</td>
<td> </td>
</tr>
<tr>
<td>2010/11/09 19:18:18</td>
<td>commSetEvents(fd=15)</td>
<td> </td>
</tr>
<tr>
<td>2010/11/09 19:18:18</td>
<td>comm_call_handlers(): got fd=15 read_event=1 write_event=0 F->read_handler=0x423370 F->write_handler=(nil)</td>
<td> </td>
</tr>
<tr>
<td>2010/11/09 19:18:18</td>
<td>comm_call_handlers(): Calling read handler on fd=15</td>
<td> </td>
</tr>
<tr>
<td>2010/11/09 19:18:18</td>
<td>clientReadRequest: FD 15: reading request…</td>
<td> </td>
</tr>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataLock: 0x9a14f82010/11/09 19:18:18</td>
<td>cbdataValid: 0x9a14f8</td>
</tr>
</tbody>
</table>
<p>3、读取请求信息,创建entry空间,如下:</p>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>Parser: retval 1: from 0->68: method 0->2; url 4->57; version 59->67 (1/0)</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>parseHttpRequest: Method is ‘GET’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>parseHttpRequest: URI is ‘http://shanzhai.china.com/images/simple/siteGuideR.gif’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>parseHttpRequest: req_hdr = {Referer: http://w.china.com</td>
</tr>
</tbody>
</table>
<p>User-Agent: Wget/1.10.2 (Red Hat modified)</p>
<p>Accept: <em>/</em></p>
<p>Host: shanzhai.china.com</p>
<p>}</p>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>parseHttpRequest: prefix_sz = 183, req_line_sz = 69</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>parseHttpRequest: Request Header is</td>
</tr>
</tbody>
</table>
<p> </p>
<p>Referer: http://w.china.com</p>
<p>User-Agent: Wget/1.10.2 (Red Hat modified)</p>
<p>Accept: <em>/</em></p>
<p>Host: shanzhai.china.com</p>
<p> </p>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>parseHttpRequest: Complete request received</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>commSetTimeout: FD 15 timeout 86400</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>init-ing hdr: 0x9604f8 owner: 1</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>parsing hdr: (0x9604f8)</td>
</tr>
</tbody>
</table>
<p>Referer: http://w.china.com</p>
<p>User-Agent: Wget/1.10.2 (Red Hat modified)</p>
<p>Accept: <em>/</em></p>
<p>Host: shanzhai.china.com</p>
<p> </p>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>creating entry 0x9a5610: near ‘Referer: http://w.china.com’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>created entry 0x9a5610: ‘Referer: http://w.china.com’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>0x9604f8 adding entry: 45 at 0</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>creating entry 0x847c10: near ‘User-Agent: Wget/1.10.2 (Red Hat modified)’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>created entry 0x847c10: ‘User-Agent: Wget/1.10.2 (Red Hat modified)’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>0x9604f8 adding entry: 50 at 1</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>creating entry 0x9a4d20: near ‘Accept: <em>/</em>’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>created entry 0x9a4d20: ‘Accept: <em>/</em>’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>0x9604f8 adding entry: 0 at 2</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>creating entry 0x9a1cb0: near ‘Host: shanzhai.china.com’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>created entry 0x9a1cb0: ‘Host: shanzhai.china.com’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>0x9604f8 adding entry: 27 at 3</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>removing 183 bytes; conn->in.offset = 0</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>0x9604f8 lookup for 20</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>clientSetKeepaliveFlag: http_ver = 1.0</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>clientSetKeepaliveFlag: method = GET</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>0x9604f8 lookup for 41</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>0x9604f8 lookup for 9</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>0x9604f8 lookup for 52</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>0x9604f8 lookup for 41</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>0x9604f8 lookup for 9</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>0x9604f8 lookup for 59</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataLock: 0x734ed8</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataLock: 0x9a14f8</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataLock: 0x9a17a8</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataValid: 0x734ed8</td>
</tr>
</tbody>
</table>
<p> </p>
<p>至此完成了请求信息的存取,并保持HTTP/1.0下的keepalive。</p>
<p> </p>
<p>4、请求acl检查部分</p>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclCheck: checking ‘http_access allow swfs !notnull_refer’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAclList: checking swfs</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAcl: checking ‘acl swfs url_regex -i ^http://flash.shanzhai.china.com/swfsimple/com/china/ceuf/map/.*.swf’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchRegex: checking ‘http://shanzhai.china.com/images/simple/siteGuideR.gif’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchRegex: looking for ‘^http://flash.shanzhai.china.com/swfsimple/com/china/ceuf/map/.*.swf’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAclList: no match, returning 0</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataLock: 0x734b68</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataUnlock: 0x734ed8</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataValid: 0x734b68</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclCheck: checking ‘http_access deny pics !notnull_refer’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAclList: checking pics</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAcl: checking ‘acl pics url_regex -i .(jpg</td>
<td>gif</td>
<td>jpeg</td>
<td>png</td>
<td>mp3</td>
<td>smi</td>
<td>wma</td>
<td>swf)$’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchRegex: checking ‘http://shanzhai.china.com/images/simple/siteGuideR.gif’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchRegex: looking for ‘.(jpg</td>
<td>gif</td>
<td>jpeg</td>
<td>png</td>
<td>mp3</td>
<td>smi</td>
<td>wma</td>
<td>swf)$’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchRegex: match ‘.(jpg</td>
<td>gif</td>
<td>jpeg</td>
<td>png</td>
<td>mp3</td>
<td>smi</td>
<td>wma</td>
<td>swf)$’ found in ‘http://shanzhai.china.com/images/simple/siteGuid</td>
</tr>
</tbody>
</table>
<p>eR.gif’</p>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAclList: checking !notnull_refer</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAcl: checking ‘acl notnull_refer referer_regex .’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchRegex: checking ‘http://w.china.com’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchRegex: looking for ‘.’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchRegex: match ‘.’ found in ‘http://w.china.com’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAclList: no match, returning 0</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataLock: 0x735308</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataUnlock: 0x734b68</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataValid: 0x735308</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclCheck: checking ‘http_access deny pics !domain_refer’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAclList: checking pics</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAcl: checking ‘acl pics url_regex -i .(jpg</td>
<td>gif</td>
<td>jpeg</td>
<td>png</td>
<td>mp3</td>
<td>smi</td>
<td>wma</td>
<td>swf)$’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchRegex: checking ‘http://shanzhai.china.com/images/simple/siteGuideR.gif’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchRegex: looking for ‘.(jpg</td>
<td>gif</td>
<td>jpeg</td>
<td>png</td>
<td>mp3</td>
<td>smi</td>
<td>wma</td>
<td>swf)$’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchRegex: match ‘.(jpg</td>
<td>gif</td>
<td>jpeg</td>
<td>png</td>
<td>mp3</td>
<td>smi</td>
<td>wma</td>
<td>swf)$’ found in ‘http://shanzhai.china.com/images/simple/siteGuid</td>
</tr>
</tbody>
</table>
<p>eR.gif’</p>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAclList: checking !domain_refer</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAcl: checking ‘acl domain_refer referer_regex -i ^http://[^/]<em>china.com ^http://124.238.253.</em>’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchRegex: checking ‘http://w.china.com’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchRegex: looking for ‘^http://[^/]*china.com’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchRegex: match ‘^http://[^/]*china.com’ found in ‘http://w.china.com’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAclList: no match, returning 0</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataLock: 0x7355a8</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataUnlock: 0x735308</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataValid: 0x7355a8</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclCheck: checking ‘http_access allow Safe_ports Domain’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAclList: checking Safe_ports</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAcl: checking ‘acl Safe_ports port 80’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAclList: checking Domain</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAcl: checking ‘acl Domain dstdomain .china.com’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchDomainList: checking ‘shanzhai.china.com’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchDomainList: ‘shanzhai.china.com’ found</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAclList: returning 1</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclCheck: match found, returning 1</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataUnlock: 0x7355a8</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclCheckCallback: answer=1</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataValid: 0x9a17a8</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>The request GET http://shanzhai.china.com/images/simple/siteGuideR.gif is ALLOWED, because it matched ‘Domain’</td>
</tr>
</tbody>
</table>
<p>对照squid.conf,发现acl检测是从上到下依次进行(一般acl都这样),且http_access后最多只能有2个aclname</p>
<p> </p>
<p>5、rewrite部分</p>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>clientRedirectStart: ‘http://shanzhai.china.com/images/simple/siteGuideR.gif’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>clientRedirectDone: ‘http://shanzhai.china.com/images/simple/siteGuideR.gif’ result=NULL</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>clientStoreURLRewriteStart: ‘http://shanzhai.china.com/images/simple/siteGuideR.gif’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>clientStoreURLRewriteDone: ‘http://shanzhai.china.com/images/simple/siteGuideR.gif’ result=NULL</td>
</tr>
</tbody>
</table>
<p> </p>
<p>6、cache配置</p>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>clientInterpretRequestHeaders: REQ_NOCACHE = NOT SET</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>clientInterpretRequestHeaders: REQ_CACHABLE = SET</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>clientInterpretRequestHeaders: REQ_HIERARCHICAL = SET</td>
</tr>
</tbody>
</table>
<p>7、cache_store检查</p>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>clientProcessRequest: GET ‘http://shanzhai.china.com/images/simple/siteGuideR.gif’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>0x9604f8 lookup for 53</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>storeGet: looking up 15EBB8A96296D7407EA1D03F075666BC</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>clientProcessRequest2: default HIT</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>clientProcessRequest: TCP_HIT for ‘http://shanzhai.china.com/images/simple/siteGuideR.gif’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>storeLockObject: (client_side.c:3544): key ‘15EBB8A96296D7407EA1D03F075666BC’ count=1</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>storeAufsDirRefObj: referencing 0x753680 0/272</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>storeLockObject: (store_client.c:122): key ‘15EBB8A96296D7407EA1D03F075666BC’ count=2</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>storeAufsDirRefObj: referencing 0x753680 0/272</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>storeClientCopy: 15EBB8A96296D7407EA1D03F075666BC, seen 0, want 0, size 4096, cb 0x41dd90, cbdata 0x9a17a8</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataLock: 0x9a17a8</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataLock: 0x9a2178</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>storeClientCopy2: 15EBB8A96296D7407EA1D03F075666BC</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>storeClientCopy3: Copying from memory</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>stmemCopy: offset 0: size 4096</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataValid: 0x9a17a8</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>clientCacheHit: http://shanzhai.china.com/images/simple/siteGuideR.gif = 200</td>
</tr>
</tbody>
</table>
<p> </p>
<p>8、过期时间</p>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>refreshCheck: ‘http://shanzhai.china.com/images/simple/siteGuideR.gif’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>FRESH: expires 1320836796 >= check_time 1289301498</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>Staleness = -1</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>refreshCheck: Matched ‘<none> 0 20% 259200'</none></td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>refreshCheck: age = 702</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td> check_time: Tue, 09 Nov 2010 11:18:18 GMT</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td> entry->timestamp: Tue, 09 Nov 2010 11:06:36 GMT</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>clientCacheHit: refreshCheckHTTPStale returned 0</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>clientCacheHit: HIT</td>
</tr>
</tbody>
</table>
<p> </p>
<p>9、继续完成entry</p>
<table>
<tbody>
<tr>
<td>……2010/11/09 19:18:18</td>
<td>destroying entry 0x9a4cc0: ‘Connection: keep-alive’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>……2010/11/09 19:18:18</td>
<td>created entry 0x9a4c60: ‘Connection: close’</td>
</tr>
</tbody>
</table>
<p> </p>
<p>10、响应acl检查部分</p>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclCheck: checking ‘http_reply_access allow all’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAclList: checking all</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAcl: checking ‘acl all src 0.0.0.0/0.0.0.0’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchIp: ‘124.238.250.28’ found</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclMatchAclList: returning 1</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclCheck: match found, returning 1</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataUnlock: 0x731988</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>aclCheckCallback: answer=1</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataValid: 0x9a17a8</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>The reply for GET http://shanzhai.china.com/images/simple/siteGuideR.gif is ALLOWED, because it matched ‘all’</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>packing sline 0x9a5010 using 0x7fffdb895d10:</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>HTTP/1.1 200 OK</td>
</tr>
</tbody>
</table>
<p> </p>
<p>11、传输文件</p>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>packing hdr: (0x9a5028)</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>comm_write: FD 15: sz 362: hndl 0x424d60: data 0x9a17a8.</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataLock: 0x9a17a8</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>commSetSelect: FD 15 type 2</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>commSetEvents(fd=15)</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataUnlock: 0x9a17a8</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataUnlock: 0x9a14f8</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataFree: 0x846f18</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataFree: Freeing 0x846f18</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>2010/11/09 19:18:18</td>
<td>cbdataUnlock: 0x9a17a8</td>
</tr>
</tbody>
</table>
<p> </p>
inotify使用一例
2010-11-04T00:00:00+08:00
linux
inotify
http://chenlinux.com/2010/11/04/intro-inotify
<p>在对一个大磁盘进行inotify监听时,爆出如下错误:</p>
<p>Failed to watch /mnt/;<br />
upper limit on inotify watches reached!<br />
Please increase the amount of inotify watches allowed per user via `/proc/sys/fs/inotify/max_user_watches’.</p>
<p>cat一下这个文件,默认值是8192;df -i看看inode数量,远远大过这个值~</p>
<p>echo 8192000 > /proc/sys/fs/inotify/max_user_watches即可~</p>
集群故障检查记录
2010-10-27T00:00:00+08:00
linux
http://chenlinux.com/2010/10/27/problem-about-vrrp-and-intro-flood-ping
<p>某服务器集群,是双lvs_keepalived+多nginx结构。最近突然发现流量监控出现较大波动,nginx的access.log时常出现持续几十秒的无外来访问情况(即只有LVS的ip过来的400探测)。</p>
<p>在两台lvs设备上的/var/log/messages上都看到大量的VIP切换记录:</p>
<p>Oct 22 09:47:12 localhost Keepalived_vrrp: VRRP_Instance(VI_10) Transition to MASTER STATE<br />
Oct 22 09:47:13 localhost Keepalived_vrrp: VRRP_Instance(VI_10) Entering MASTER STATE<br />
Oct 22 09:47:13 localhost Keepalived_vrrp: VRRP_Instance(VI_10) setting protocol VIPs.</p>
<p>Oct 22 09:47:14 localhost Keepalived_vrrp: VRRP_Instance(VI_10) Received higher prio advert<br />
Oct 22 09:47:14 localhost Keepalived_vrrp: VRRP_Instance(VI_10) Entering BACKUP STATE<br />
Oct 22 09:47:14 localhost Keepalived_vrrp: VRRP_Instance(VI_10) removing protocol VIPs.</p>
<p>相互之间ping丢包相当严重,而ping同一网段其他ip都没问题。</p>
<p>学来一个比较直观而且细腻的ping用法:ping -f $IP -c 5000</p>
<p>man上对-f的解释如下:</p>
<p>-f Flood ping. For every ECHO_REQUEST sent a period “.” is printed, while for ever ECHO_REPLY received a backspace is printed. This provides a rapid display of how many packets are being dropped. If interval is not given, it sets interval to zero and outputs packets as fast as they come back or one hundred times per second, whichever is more. Only the superuser may use this option with zero interval.</p>
<p>-f 洪水ping,每当发送一个ECHO_REQUEST请求时,都在屏幕上打印一个点”.”,而收到ECHO_REPLY时就退格一次。以此简洁明了的显示出丢了多少个包。如果不明确指定发包间隔,默认间隔时间为0,即收包有多快,发包就有多快,可能每秒100次,或许更快~注意:只有超级用户root才能不设间隔的使用洪水ping参数-f</p>
<p>我们把这个丢包.叫做贪吃蛇,哈哈~~</p>
<p>采用更换服务器网线、强制指定网卡速率等办法,丢包率从近30%下降到了5%,问题依然没有全部解决。最后重启机器,就OK了……真实问题是否和ipvs有关,待查ing</p>
Myisamchk小工具使用手册(转)
2010-10-27T00:00:00+08:00
database
MySQL
http://chenlinux.com/2010/10/27/intro-myisamchk
<p>前不久碰上mysql的表损坏,百度到这篇文章,现在转载过来,原文出处:<a href="http://logzgh.itpub.net/post/3185/454455">http://logzgh.itpub.net/post/3185/454455</a></p>
<p>Myisamchk是MyISAM表维护的一个非常实用的工具。可以使用myisamchk实用程序来获得有关数据库表的信息或检查、修复、优化他们。myisamchk适用MyISAM表(对应.MYI和.MYD文件的表)。<br />
1.myisamchk的调用方法<br />
myisamchk [options] tbl_name …<br />
其中options指定你想让myisamchk干什么。</p>
<p>它允许你通过使用模式“*.MYI”指定在一个目录所有的表。<br />
shell> myisamchk *.MYI</p>
<p>推荐的快速检查所有MyISAM表的方式是:</p>
<p>shell> myisamchk –silent –fast /path/to/datadir/<em>/</em>.MYI<br />
当你运行myisamchk时,必须确保其它程序不使用表。</p>
<p>当你运行myisamchk时内存分配重要.MYIsamchk使用的内存大小不能超过用-O选项指定的。对于大多数情况,使用-O sort=16M应该足够了。<br />
另外在修复时myisamchk需要大量硬盘空间,基本上是所涉及表空间的双倍大小。</p>
<p>2.myisamchk的一般选项<br />
–debug=debug_options, -# debug_options<br />
输出调试记录文件。debug_options字符串经常是’d:t:o,filename’。</p>
<p>–silent,-s<br />
沉默模式。仅当发生错误时写输出。</p>
<p>–wait, -w<br />
如果表被锁定,不是提示错误终止,而是在继续前等待到表被解锁。<br />
如果不使用–skip-external-locking,可以随时使用myisamchk来检查表。当检查表时,所有尝试更新表的客户端将等待,直到myisamchk准备好可以继续。<br />
请注意如果用–skip-external-locking选项运行mysqld,只能用另一个myisamchk命令锁定表。</p>
<p>–var_name=value<br />
可以通过–var_name=value选项设置下面的变量:<br />
decode_bits 9<br />
ft_max_word_len 取决于版本<br />
ft_min_word_len 4<br />
ft_stopword_file 内建列表<br />
key_buffer_size 523264<br />
myisam_block_size 1024<br />
read_buffer_size 262136<br />
sort_buffer_size 2097144<br />
sort_key_blocks 16<br />
stats_method nulls_unequal<br />
write_buffer_size 262136<br />
如果想要快速修复,将key_buffer_size和sort_buffer_size变量设置到大约可用内存的25%。<br />
可以将两个变量设置为较大的值,因为一个时间只使用一个变量。<br />
myisam_block_size是用于索引块的内存大小。<br />
stats_method影响当给定–analyze选项时,如何为索引统计搜集处理NULL值。</p>
<p>3.myisamchk的检查选项<br />
–check, -c<br />
检查表的错误。如果你不明确指定操作类型选项,这就是默认操作。</p>
<p>–check-only-changed, -C<br />
只检查上次检查后有变更的表。</p>
<p>–extend-check, -e<br />
非常仔细地检查表。如果表有许多索引将会相当慢。</p>
<p>–fast,-F<br />
只检查没有正确关闭的表。</p>
<p>–force, -f<br />
如果myisamchk发现表内有任何错误,则自动进行修复。</p>
<p>–information, -i<br />
打印所检查表的统计信息。</p>
<p>–medium-check, -m<br />
比–extend-check更快速地进行检查。只能发现99.99%的错误</p>
<p>–update-state, -U<br />
将信息保存在.MYI文件中,来表示表检查的时间以及是否表崩溃了。该选项用来充分利用–check-only-changed选项,<br />
但如果mysqld服务器正使用表并且正用–skip-external-locking选项运行时不应使用该选项。</p>
<p>–read-only, -T<br />
不要将表标记为已经检查。如果你使用myisamchk来检查正被其它应用程序使用而没有锁定的表很有用</p>
<p>4.myisamchk的修复选项<br />
–backup, -B<br />
将.MYD文件备份为file_name-time.BAK</p>
<p>–character-sets-dir=path<br />
字符集安装目录。</p>
<p>–correct-checksum<br />
纠正表的校验和信息。</p>
<p>–data-file-length=len, -D len<br />
数据文件的最大长度</p>
<p>–extend-check,-e<br />
进行修复,试图从数据文件恢复每一行。一般情况会发现大量的垃圾行。不要使用该选项,除非你不顾后果。</p>
<p>–force, -f<br />
覆盖旧的中间文件(文件名类似tbl_name.TMD),而不是中断</p>
<p>–keys-used=val, -k val<br />
对于myisamchk,该选项值为位值,说明要更新的索引。选项值的每一个二进制位对应表的一个索引,其中第一个索引对应位0。<br />
选项值0禁用对所有索引的更新,可以保证快速插入。通过myisamchk -r可以重新激活被禁用的索引。</p>
<p>–parallel-recover, -p<br />
与-r和-n的用法相同,但使用不同的线程并行创建所有键。</p>
<p>–quick,-q<br />
不修改数据文件,快速进行修复。</p>
<p>–recover, -r<br />
可以修复几乎所有一切问题,除非唯一的键不唯一时(对于MyISAM表,这是非常不可能的情况)。如果你想要恢复表,<br />
这是首先要尝试的选项。如果myisamchk报告表不能用-r恢复,则只能尝试-o。<br />
在不太可能的情况下-r失败,数据文件保持完好)。</p>
<p>–safe-recover, -o<br />
使用一个老的恢复方法读取,按顺序读取所有行,并根据找到的行更新所有索引树。这比-r慢些,<br />
但是能处理-r不能处理的情况。该恢复方法使用的硬盘空间比-r少。一般情况,你应首先用-r维修,如果-r失败则用-o。</p>
<p>–sort-recover, -n<br />
强制myisamchk通过排序来解析键值,即使临时文件将可能很大。</p>
<p>5.myisamchk的其他选项<br />
myisamchk支持以下表检查和修复之外的其它操作的选项:</p>
<p>–analyze,-a<br />
分析键值的分布。这通过让联结优化器更好地选择表应该以什么次序联结和应该使用哪个键来改进联结性能。<br />
要想获取分布相关信息,使用myisamchk –description –verbose tbl_name命令或SHOW KEYS FROM tbl_name语句。</p>
<p>–sort-index, -S<br />
以从高到低的顺序排序索引树块。这将优化搜寻并且将使按键值的表扫描更快。</p>
<p>–set-auto-increment[=value], -A[value]<br />
强制从给定值开始的新记录使用AUTO_INCREMENT编号(或如果已经有AUTO_INCREMENT值大小的记录,应使用更高值)。<br />
如果未指定value,新记录的AUTO_INCREMENT编号应使用当前表的最大值加上1。</p>
<p>–description, -d<br />
打印出关于表的描述性信息。<br />
例如:<br />
[root@qa-sandbox-1 mysql]# myisamchk -d user.MYI<br />
MyISAM file: user.MYI<br />
Record format: Packed<br />
Character set: latin1_swedish_ci (8)<br />
Data records: 6 Deleted blocks: 1<br />
Recordlength: 346</p>
<p>table description:<br />
Key Start Len Index Type<br />
1 1 180 unique char packed stripped<br />
181 48 char stripped</p>
<p>6.如何修复表</p>
<p>检查你的表<br />
如果你有很多时间,运行myisamchk *.MYI或myisamchk -e *.MYI。使用-s(沉默)选项禁止不必要的信息。<br />
如果mysqld服务器处于宕机状态,应使用–update-state选项来告诉myisamchk将表标记为’检查过的’。</p>
<p>简单安全的修复<br />
首先,试试myisamchk -r -q tbl_name(-r -q意味着“快速恢复模式”)<br />
如果在修复时,你得到奇怪的错误(例如out of memory错误),或如果myisamchk崩溃,到阶段3。</p>
<p>困难的修复<br />
只有在索引文件的第一个16K块被破坏,或包含不正确的信息,或如果索引文件丢失,你才应该到这个阶段。在这种情况下,需要创建一个新的索引文件。按如下步骤操做:</p>
<ol>
<li>把数据文件移到安全的地方。</li>
<li>使用表描述文件创建新的(空)数据文件和索引文件:</li>
<li>shell> mysql db_name</li>
<li>mysql> SET AUTOCOMMIT=1;</li>
<li>mysql> TRUNCATE TABLE tbl_name;</li>
<li>mysql> quit<br />
如果你的MySQL版本没有TRUNCATE TABLE,则使用DELETE FROM tbl_name。</li>
<li>将老的数据文件拷贝到新创建的数据文件之中。(不要只是将老文件移回新文件之中;你要保留一个副本以防某些东西出错。)</li>
</ol>
<p>回到阶段2。现在myisamchk -r -q应该工作了。(这不应该是一个无限循环)。</p>
<p>你还可以使用REPAIR TABLE tbl_name USE_FRM,将自动执行整个程序。</p>
<p>非常困难的修复<br />
只有.frm描述文件也破坏了,你才应该到达这个阶段。这应该从未发生过,因为在表被创建以后,描述文件就不再改变了。</p>
<ol>
<li>从一个备份恢复描述文件然后回到阶段3。你也可以恢复索引文件然后回到阶段2。对后者,你应该用myisamchk -r启动。</li>
<li>如果你没有进行备份但是确切地知道表是怎样创建的,在另一个数据库中创建表的一个拷贝。删除新的数据文件,然后从其他数据库将描述文件和索引文件移到破坏的数据库中。这样提供了新的描述和索引文件,但是让.MYD数据文件独自留下来了。回到阶段2并且尝试重建索引文件。</li>
</ol>
<p>7.清理碎片<br />
对Innodb 表则可以通过执行以下语句来整理碎片,提高索引速度:<br />
ALTER TABLE tbl_name ENGINE = Innodb;<br />
这其实是一个 NULL 操作,表面上看什么也不做,实际上重新整理碎片了。</p>
<p>对myisam表格,为了组合碎片记录并且消除由于删除或更新记录而浪费的空间,以恢复模式运行myisamchk:</p>
<p>shell> myisamchk -r tbl_name</p>
<p>你可以用SQL的OPTIMIZE TABLE语句使用的相同方式来优化表,OPTIMIZE TABLE可以修复表并对键值进行分析,并且可以对索引树进行排序以便更快地查找键值。</p>
<p>8.建立表检查计划<br />
运行一个crontab,每天定期检查所有的myisam表格。<br />
35 0 * * 0 /path/to/myisamchk –fast –silent /path/to/datadir/<em>/</em>.MYI</p>
<p>9.获取表的信息</p>
<p>myisamchk -d tbl_name:以“描述模式”运行myisamchk,生成表的描述<br />
myisamchk -d -v tbl_name: 为了生成更多关于myisamchk正在做什么的信息,加上-v告诉它以冗长模式运行。<br />
myisamchk -eis tbl_name:仅显示表的最重要的信息。因为必须读取整个表,该操作很慢。<br />
myisamchk -eiv tbl_name:这类似 -eis,只是告诉你正在做什么。</p>
<p>10.Myisamchk产生的信息解释</p>
<p>MyISAM file<br />
ISAM(索引)文件名。</p>
<p>File-version<br />
ISAM格式的版本。当前总是2。</p>
<p>Creation time<br />
数据文件创建的时间。</p>
<p>Recover time<br />
索引/数据文件上次被重建的时间。</p>
<p>Data records<br />
在表中有多少记录。</p>
<p>Deleted blocks<br />
有多少删除的块仍然保留着空间。你可以优化表以使这个空间减到最小。参见第7章:优化。</p>
<p>Datafile parts<br />
对动态记录格式,这指出有多少数据块。对于一个没有碎片的优化过的表,这与Data records相同。</p>
<p>Deleted data<br />
不能回收的删除数据有多少字节。你可以优化表以使这个空间减到最小。参见第7章:优化。</p>
<p>Datafile pointer<br />
数据文件指针的大小,以字节计。它通常是2、3、4或5个字节。大多数表用2个字节管理,但是目前这还不能从MySQL控制。<br />
对固定表,这是一个记录地址。对动态表,这是一个字节地址。</p>
<p>Keyfile pointer<br />
索引文件指针的大小,以字节计。它通常是1、2或3个字节。大多数表用 2 个字节管理,但是它自动由MySQL计算。<br />
它总是一个块地址。</p>
<p>Max datafile length<br />
表的数据文件(.MYD文件)能够有多长,以字节计。</p>
<p>Max keyfile length<br />
表的键值文件(.MYI文件)能够有多长,以字节计。</p>
<p>Recordlength<br />
每个记录占多少空间,以字节计。</p>
<p>Record format<br />
用于存储表行的格式。上面的例子使用Fixed length。其他可能的值是Compressed和Packed。</p>
<p>table description<br />
在表中所有键值的列表。对每个键,给出一些底层的信息:<br />
Key<br />
该键的编号。<br />
Start<br />
该索引部分从记录的哪里开始。<br />
Len<br />
该索引部分是多长。对于紧凑的数字,这应该总是列的全长。对字符串,它可以比索引的列的全长短些,<br />
因为你可能会索引到字符串列的前缀。<br />
Index<br />
unique或multip(multiple)。表明一个值是否能在该索引中存在多次。<br />
Type<br />
该索引部分有什么数据类型。这是一个packed、stripped或empty选项的ISAM数据类型。<br />
Root<br />
根索引块的地址。<br />
Blocksize<br />
每个索引块的大小。默认是1024,但是从源码构建MySQL时,该值可以在编译时改变。<br />
Rec/key<br />
这是由优化器使用的统计值。它告诉对该键的每个值有多少条记录。唯一键总是有一个1值。<br />
在一个表被装载后(或变更很大),可以用myisamchk -a更新。如果根本没被更新,给定一个30的默认值。<br />
在上面例子的表中,第9个键有两个table description行。这说明它是有2个部分的多部键。</p>
<p>Keyblocks used<br />
键块使用的百分比是什么。当在例子中使用的表刚刚用myisamchk重新组织时,该值非常高(很接近理论上的最大值)。</p>
<p>Packed<br />
MySQL试图用一个通用后缀压缩键。这只能被用于CHAR/VARCHAR/DECIMAL列的键。对于左部分类似的长字符串,<br />
能显著地减少使用空间。在上面的第3个例子中,第4个键是10个字符长,可以减少60%的空间。</p>
<p>Max levels<br />
对于该键的B树有多深。有长键的大表有较高的值。</p>
<p>Records<br />
表中有多少行。</p>
<p>M.recordlength<br />
平均记录长度。对于有定长记录的表,这是准确的记录长度,因为所有记录的长度相同。</p>
<p>Packed<br />
MySQL从字符串的结尾去掉空格。Packed值表明这样做达到的节约的百分比。</p>
<p>Recordspace used<br />
数据文件被使用的百分比。</p>
<p>Empty space<br />
数据文件未被使用的百分比。</p>
<p>Blocks/Record<br />
每个记录的平均块数(即,一个碎片记录由多少个连接组成)。对固定格式表,这总是1。该值应该尽可能保持接近1.0。<br />
如果它变得太大,你可以重新组织表。参见第7章:优化。</p>
<p>Recordblocks<br />
多少块(链接)被使用。对固定格式,它与记录的个数相同。</p>
<p>Deleteblocks<br />
多少块(链接)被删除。</p>
<p>Recorddata<br />
在数据文件中使用了多少字节。</p>
<p>Deleted data<br />
在数据文件中多少字节被删除(未使用)。</p>
<p>Lost space<br />
如果一个记录被更新为更短的长度,就损失了一些空间。这是所有这样的损失之和,以字节计。</p>
<p>Linkdata<br />
当使用动态表格式,记录碎片用指针连接(每个4 ~ 7字节)。 Linkdata指这样的指针使用的内存量之和。</p>
mysql错误日志中文翻译
2010-10-21T00:00:00+08:00
database
MySQL
http://chenlinux.com/2010/10/21/translation-of-mysql-error-log
<p>1005:创建表失败</p>
<p>1006:创建数据库失败</p>
<p>1007:数据库已存在,创建数据库失败</p>
<p>1008:数据库不存在,删除数据库失败</p>
<p>1009:不能删除数据库文件导致删除数据库失败</p>
<p>1010:不能删除数据目录导致删除数据库失败</p>
<p>1011:删除数据库文件失败</p>
<p>1012:不能读取系统表中的记录</p>
<p>1016: 无法打开文件</p>
<p>1020:记录已被其他用户修改</p>
<p>1021:硬盘剩余空间不足,请加大硬盘可用空间</p>
<p>1022:关键字重复,更改记录失败</p>
<p>1023:关闭时发生错误</p>
<p>1024:读文件错误</p>
<p>1025:更改名字时发生错误</p>
<p>1026:写文件错误</p>
<p>1032:记录不存在</p>
<p>1036:数据表是只读的,不能对它进行修改</p>
<p>1037:系统内存不足,请重启数据库或重启服务器</p>
<p>1038:用于排序的内存不足,请增大排序缓冲区</p>
<p>1040:已到达数据库的最大连接数,请加大数据库可用连接数</p>
<p>1041:系统内存不足</p>
<p>1042:无效的主机名</p>
<p>1043:无效连接</p>
<p>1044:当前用户没有访问数据库的权限</p>
<p>1045:不能连接数据库,用户名或密码错误</p>
<p>1048:字段不能为空</p>
<p>1049:数据库不存在</p>
<p>1050:数据表已存在</p>
<p>1051:数据表不存在</p>
<p>1054:字段不存在</p>
<p>1062:字段值重复,入库失败</p>
<p>1065:无效的SQL语句,SQL语句为空</p>
<p>1081:不能建立Socket连接</p>
<p>1114:数据表已满,不能容纳任何记录</p>
<p>1116:打开的数据表太多</p>
<p>1129:数据库出现异常,请重启数据库</p>
<p>1130:连接数据库失败,没有连接数据库的权限</p>
<p>1133:数据库用户不存在</p>
<p>1141:当前用户无权访问数据库</p>
<p>1142:当前用户无权访问数据表</p>
<p>1143:当前用户无权访问数据表中的字段</p>
<p>1146:数据表不存在</p>
<p>1147:未定义用户对数据表的访问权限</p>
<p>1149:SQL语句语法错误</p>
<p>1158:网络错误,出现读错误,请检查网络连接状况</p>
<p>1159:网络错误,读超时,请检查网络连接状况</p>
<p>1160:网络错误,出现写错误,请检查网络连接状况</p>
<p>1161:网络错误,写超时,请检查网络连接状况</p>
<p>1169:字段值重复,更新记录失败</p>
<p>1177:打开数据表失败</p>
<p>1180:提交事务失败</p>
<p>1181:回滚事务失败</p>
<p>1203:当前用户和数据库建立的连接已到达数据库的最大连接数,请增大可用的数据库连接数或重启数据库</p>
<p>1205:加锁超时</p>
<p>1211:当前用户没有创建用户的权限</p>
<p>1216:外键约束检查失败,更新子表记录失败</p>
<p>1217:外键约束检查失败,删除或修改主表记录失败</p>
<p>1226:当前用户使用的资源已超过所允许的资源,请重启数据库或重启服务器</p>
<p>1227:权限不足,您无权进行此操作</p>
<p>1235:MySQL版本过低,不具有本功能</p>
<p>1250:客户端不支持服务器要求的认证协议,请考虑升级客户端</p>
页面自动刷新小问题
2010-10-21T00:00:00+08:00
monitor
cacti
http://chenlinux.com/2010/10/21/auto-refresh-of-pnp4nagios
<p>cacti的页面,每5分钟自动刷新一次,这样就可以“实时”的看到rrd绘图的最新结果。</p>
<p>nagios加上pnp插件后,也有rrd绘图的页面,但却没有自动刷新。</p>
<p>查看pnp/config.cfg,里面已经配置了$conf[‘refresh’] = “90”;但页面没有反应。</p>
<p>于是分别查看cacti和pnp的页面源代码,cacti的如下:</p>
<p><meta http-equiv=refresh content=’300’></p>
<p>pnp的如下:</p>
<meta http-equiv="refresh" content="90; URL=***" />
<p>看来时这个META标签写法不对,修改nagios/share/pnp/include/funcation.inc.php如下:</p>
<ul>
<li>
<p>print “<meta http-equiv="refresh" content="” . $conf[‘refresh’] . “; URL=” . $_SERVER[‘REQUEST_URI’] . “">\n”;</p>
</li>
<li>
<p>print “<meta http-equiv="refresh" content="”.$conf[‘refresh’].”">\n”;</p>
</li>
</ul>
<p>保存退出,刷新页面后等待90s,果然更新了~~</p>
tcp overflow报错
2010-10-20T00:00:00+08:00
linux
http://chenlinux.com/2010/10/20/solution-of-tcp_overflow
<p>在对TCP参数进行sysctl优化时,通常会减小net.ipv4.tcp_max_tw_buckets这个设置,以减少服务器的TIME_WAIT数量,提高服务器响应速度。</p>
<p>不过对于squid服务器来说,这个优化没什么作用,而且在/var/log/messages和dmesg中大屏大屏的出现如下kernel报错。很是烦人……<br />
printk: 7277 messages suppressed.<br />
TCP: time wait bucket table overflow</p>
<p>更深一步,如果去linux内核代码中搜索的话,可以发现如下:</p>
<p>struct inet_timewait_sock *tw = NULL;<br />
const struct inet_connection_sock *icsk = inet_csk(sk);<br />
const struct tcp_sock *tp = tcp_sk(sk);<br />
int recycle_ok = 0;</p>
<p>if (tcp_death_row.sysctl_tw_recycle && tp->rx_opt.ts_recent_stamp)<br />
recycle_ok = icsk->icsk_af_ops->remember_stamp(sk);</p>
<p>if (tcp_death_row.tw_count < tcp_death_row.sysctl_max_tw_buckets)<br />
tw = inet_twsk_alloc(sk, state);</p>
<p>if (tw != NULL) {</p>
<p>……</p>
<p>} else {</p>
<p>/* Sorry, if we’re out of memory, just CLOSE this<br />
* socket up. We’ve got bigger problems than<br />
* non-graceful socket closings.<br />
*/<br />
LIMIT_NETDEBUG(KERN_INFO “TCP: time wait bucket table overflow\n”);<br />
}</p>
<p>……</p>
<p>struct inet_timewait_sock *inet_twsk_alloc(const struct sock *sk, const int state)<br />
{<br />
struct inet_timewait_sock *tw =<br />
kmem_cache_alloc(sk->sk_prot_creator->twsk_prot->twsk_slab,<br />
GFP_ATOMIC);<br />
if (tw != NULL) {</p>
<p>……</p>
<p>}<br />
return tw;<br />
}</p>
<p>也就是,报出该错有两个可能性:</p>
<p>1、tcp_death_row.tw_count >= tcp_death_row.sysctl_max_tw_buckets</p>
<p>2、调用 kmem_cache_alloc 分配 tw 错误</p>
<p>所以,对squid服务器,可以适当放大net.ipv4.tcp_max_tw_buckets的设置~~</p>
nagios性能监控
2010-10-19T00:00:00+08:00
monitor
nagios
http://chenlinux.com/2010/10/19/intro-nagiostats
<p>nagios自带有性能监控工具nagiostats,安装在nagios路径的bin/下。直接运行即可看到主机、服务的检测频率,故障数量及比例等。举例如下:</p>
<p>Nagios Stats 3.0.3<br />
Copyright (c) 2003-2008 Ethan Galstad (www.nagios.org)<br />
Last Modified: 06-25-2008<br />
License: GPL</p>
<h2 id="current-status-data">CURRENT STATUS DATA</h2>
<p>Status File: /usr/local/nagios/var/status.dat<br />
Status File Age: 0d 0h 0m 9s<br />
Status File Version: 3.0.3</p>
<p>Program Running Time: 6d 17h 48m 34s<br />
Nagios PID: 14400<br />
Used/High/Total Command Buffers: 0 / 0 / 4096</p>
<p>Total Services: 1343<br />
Services Checked: 1343<br />
Services Scheduled: 1343<br />
Services Actively Checked: 1343<br />
Services Passively Checked: 0<br />
Total Service State Change: 0.000 / 11.580 / 0.009 %<br />
Active Service Latency: 0.001 / 4.541 / 0.451 sec<br />
Active Service Execution Time: 0.007 / 6.723 / 0.202 sec<br />
Active Service State Change: 0.000 / 11.580 / 0.009 %<br />
Active Services Last 1/5/15/60 min: 379 / 927 / 1178 / 1343<br />
Passive Service Latency: 0.000 / 0.000 / 0.000 sec<br />
Passive Service State Change: 0.000 / 0.000 / 0.000 %<br />
Passive Services Last 1/5/15/60 min: 0 / 0 / 0 / 0<br />
Services Ok/Warn/Unk/Crit: 1343 / 0 / 0 / 0<br />
Services Flapping: 0<br />
Services In Downtime: 0</p>
<p>Total Hosts: 239<br />
Hosts Checked: 239<br />
Hosts Scheduled: 239<br />
Hosts Actively Checked: 239<br />
Host Passively Checked: 0<br />
Total Host State Change: 0.000 / 0.000 / 0.000 %<br />
Active Host Latency: 0.014 / 5.048 / 2.766 sec<br />
Active Host Execution Time: 4.006 / 5.095 / 4.018 sec<br />
Active Host State Change: 0.000 / 0.000 / 0.000 %<br />
Active Hosts Last 1/5/15/60 min: 223 / 238 / 239 / 239<br />
Passive Host Latency: 0.000 / 0.000 / 0.000 sec<br />
Passive Host State Change: 0.000 / 0.000 / 0.000 %<br />
Passive Hosts Last 1/5/15/60 min: 0 / 0 / 0 / 0<br />
Hosts Up/Down/Unreach: 239 / 0 / 0<br />
Hosts Flapping: 0<br />
Hosts In Downtime: 0</p>
<p>Active Host Checks Last 1/5/15 min: 183 / 444 / 1135<br />
Scheduled: 183 / 444 / 1133<br />
On-demand: 0 / 0 / 2<br />
Parallel: 183 / 444 / 1134<br />
Serial: 0 / 0 / 0<br />
Cached: 0 / 0 / 1<br />
Passive Host Checks Last 1/5/15 min: 0 / 0 / 0<br />
Active Service Checks Last 1/5/15 min: 494 / 1082 / 3209<br />
Scheduled: 494 / 1082 / 3209<br />
On-demand: 0 / 0 / 0<br />
Cached: 0 / 0 / 0<br />
Passive Service Checks Last 1/5/15 min: 0 / 0 / 0</p>
<p>External Commands Last 1/5/15 min: 0 / 0 / 0</p>
<p> </p>
php编译小问题
2010-10-13T00:00:00+08:00
php
http://chenlinux.com/2010/10/13/a-problem-of-php-compiling
<p>昨天应用调整,尝试将服务器上的php降级,从5.3.0降到5.2.10,在编译时爆出如下错误提示:</p>
<p>./.libs/libgd.so: undefined reference to `png_check_sig’<br />
collect2: ld returned 1 exit status</p>
<p>因为之前有过安装,所以可以确认libpng是已经存在的。百度后看到如下说法:</p>
<p>libpng-1.4.0源码中的libpng-1.4.0.txt有说明,已经取消了png_check_sig这个函数,改用png_sig_cmp代替.自从libpng-0.90就已经反对使用png_check_sig函数了</p>
<p>所以修改php源码中的ext/gd/libgd/gd_png.c如下:</p>
<ul>
<li> if (!png_check_sig (sig, 8)) { /* bad signature */</li>
<li>if (png_sig_cmp (sig, 0, 8)) { /* bad signature */</li>
</ul>
<p>保存后重新编译,顺利通过。</p>
linux更改时区
2010-10-09T00:00:00+08:00
linux
http://chenlinux.com/2010/10/09/change-time-zone-of-linux
<p>linux上的时间,一般用定时ntpdate或者守护ntpd服务来保持正确。不过有时候会发现系统时间显示不是我们熟悉的CST,而是莫名其妙的其他地方。比如EDT什么的,ntpdate的时候,可不会自己辨别时区的~~<br />
那么就要自己手动更改了。<br />
办法很多,第一:<br />
/usr/bin/tzselect命令,然后采用一问一答的方式完成配置,这个命令其实就是一个shell脚本,利用select和case命令完成交互,从/usr/share/zoneinfo/中获取指定的文件完成操作。<br />
第二:<br />
既然知道了tzselect的操作过程,也就可以自己来干这件事情:直接进入/usr/share/zoneinfo目录,找到需要的文件,比如cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime即可;<br />
第三:<br />
各linux发行版都会有一些自己定制的配置工具,最有名的比如红帽的setup~<br />
对于时区设置,也有这种工具,redhat系列有timeconfig,debian系列有dpkg-reconfigure tzdata。</p>
google校招笔试题
2010-10-06T00:00:00+08:00
http://chenlinux.com/2010/10/06/a-question-of-google
<p>刚在CU看到传说中的一道google2011年校招笔试题,如下:</p>
<p>现在北京有一套房子,价格200万,假设房价每年上涨 10%,一个软件工程师每年固定能赚40万。如果他想买这套房子,不贷 款,不涨工资,没有其他收入,每年不吃不喝不消费,那么他需要几年才能攒够钱买这套房子?<br />
A, 5年<br />
B, 7年<br />
C, 8年<br />
D, 9年<br />
E, 永远买不起</p>
<p>最简单的思路,算算每年的房价和攒下的工资总数,最多到房价超过400万的时候就不用算了,因为那时候10%就大于工资了……</p>
<p>#!/bin/bash<br />
fangzi=200<br />
gongzi=40<br />
while true<br />
do<br />
fangzi_new=<code class="highlighter-rouge">echo $fangzi|awk '{print $1*1.1}'</code><br />
gongzi_new=<code class="highlighter-rouge">echo $gongzi|awk '{print $1+40}'</code><br />
echo -ne “$fangzi_new\t$gongzi_new\n”<br />
fangzi=<code class="highlighter-rouge">echo $fangzi_new</code><br />
gongzi=<code class="highlighter-rouge">echo $gongzi_new</code><br />
sleep 5<br />
done<br />
运行结果如下:<br />
220 80<br />
242 120<br />
266.2 160<br />
292.82 200<br />
322.102 240<br />
354.312 280<br />
389.743 320<br />
428.717 360</p>
<p>完蛋了,永远买不起……</p>
<p>不过有人提供另一种思路,如果先买40万的房子,第二年卖出,然后买84万的……</p>
<p>#!/bin/bash<br />
fangzi=200<br />
gongzi=40<br />
while true<br />
do<br />
fangzi_new=<code class="highlighter-rouge">echo $fangzi|awk '{print $1*1.1}'</code><br />
gongzi_new=<code class="highlighter-rouge">echo $gongzi|awk '{print $1*1.1+40}'</code><br />
echo -ne “$fangzi_new\t$gongzi_new\n”<br />
fangzi=<code class="highlighter-rouge">echo $fangzi_new</code><br />
gongzi=<code class="highlighter-rouge">echo $gongzi_new</code><br />
sleep 5<br />
done<br />
运行结果如下:<br />
220 84<br />
242 132.4<br />
266.2 185.64<br />
292.82 244.204<br />
322.102 308.624<br />
354.312 379.486</p>
<p>第七年就搞定了!!而第七年的工资总数是280万,倒房能倒到379.5万!!</p>
<p>唉~~</p>
perlbal
2010-09-26T00:00:00+08:00
perl
http://chenlinux.com/2010/09/26/perlbal
<p>看mogileFS,其中用到了perlbal,这是一个用perl完成的超轻量级服务器程序。包括了web、proxy、loadbalance等功能,嗯,看起来和nginx很像。<br />
又想到曾经在CU上看到一个说法,用解释性脚本语言编写的服务器程序执行相应的网页,效率最高。<br />
我就会凑点perl,下perlbal来体验一把吧~~</p>
<p>发挥一下perl的优势,直接采用CPAN安装。(从扶凯那学来cpanm方式,确实方便)<br />
<code class="highlighter-rouge">bash
wget http://xrl.us/cpanm -O /sbin/cpanm
chmod +x !$
!$ Perlbal
perlbal --help
Usage: perlbal [OPTS]
--help This usage info
--version Print perlbal release version
--config=[file] Specify Perlbal config file
(default: /etc/perlbal/perlbal.conf)
--daemon Daemonize
</code><br />
多简单呀~~<br />
不过问题来了,这个配置文件到哪找去呀~~不管是cpan.org还是danga.com/perlbal都简单到极致,压根没提及配置文件,仿佛大家都能把源码看一遍似的……<br />
好在随后在mogileFS的<a href="http://www.livejournal.com/doc/server/index.html" target="_blank">教程</a>里看到了perlbal的一些配置步骤。然后发现这个cpanm太偷懒了也不好……还是svn下来全部文件比较合适~~</p>
<div class="highlighter-rouge"><pre class="highlight"><code>svn checkout http://code.sixapart.com/svn/perlbal/trunk/
ls trunk/conf/
echoservice.conf load-balancer.conf nodelist.dat not-modified-plugin.conf perlbal.conf ssl.conf virtual-hosts.conf webserver.conf
</code></pre>
</div>
<p>各种配置文件都列出来了~<br />
不过再去trunk/lib/Perlbal/下看看pm,就发现还有很多功能没有conf出来呢,比如Stats.pm、Cache.pm、ReproxyManager.pm等等~~</p>
squid回源选择配置小结
2010-09-26T00:00:00+08:00
squid
http://chenlinux.com/2010/09/26/configure-transfer-to-peer-in-squid
<p>一共关系到cache_peer/always_direct/never_direct/hierarchy_stoplist/prefer_direct等配置项。</p>
<p>squid的<a href="http://www.deckle.co.za/squid-users-guide/Cache_Hierarchies#The_always_direct_and_never_direct_tags" target="_blank">使用指南</a>上,关于always_direct和never_direct这么写到:</p>
<p>Squid checks all <em>always_direct</em> tags before it checks any <em>never_direct</em> tags. If a matching <em>always_direct</em> tag is found, Squid will not check the <em>never_direct</em> tags, but decides which cache to talk to immediately….<br />
If the line used the <em>deny</em> keyword instead of <em>allow</em>, Squid would have simply skipped on to checking the <em>never_direct</em> lines</p>
<p>squid的<a href="http://http://www.squid-cache.org/Doc/config/hierarchy_stoplist/" target="_blank">配置说明</a>上,关于hierarchy_stoplist这么写到:</p>
<p>use this to not query neighbor caches for certain objects….never_direct overrides this option</p>
<p>squid的<a href="http://www.squid-cache.org/mail-archive/squid-users/201009/0330.html" target="_blank">邮件列表</a>上,Amos这么解释:</p>
<p>always_direct <em>prevents</em> peers being used. It does not force them. “ hierarchy_stoplist ? “ is the directive preventing the peer being used.</p>
<p>看起来挺让人晕头转向的。<br />
其实就是说:</p>
<p>always_direct allow的优先级高于never_direct,但deny(包括allow !)时则不。 <br />
hierarchy_stoplist强制请求通过域名解析回源,但never_direct又优先于它。 <br />
prefer_direct用于所有cache_peer都down了时,never_direct会报错,而prefer会转入dns解析。</p>
条件判断命令小细节
2010-09-17T00:00:00+08:00
bash
http://chenlinux.com/2010/09/17/conditional-command-details
<p>服务器上有个定时任务,执行结果输出到out.txt中以便查看,类似下行这样的:</p>
<p>*/5 * * * * /shell/runscript.sh » /shell/out.txt</p>
<p>runscript.sh中执行script.sh,完成后删除。</p>
<p>/shell目录下的所有文件不定期的通过rsync –delete同步过去。<br />
结果发现如果script在5分钟内无法执行完毕的话,cron会重复执行……<br />
考虑到rsync过来的时候out.txt已经被删除掉了决定在runscript.sh前加上一段[[ -f out.txt ]] && exit<br />
结果发现script一直无法运行了。<br />
想想原来是cron每5分钟都会去生成out.txt,只不过当空闲的时候,这个out.txt是空文件而已。<br />
修改判断为[[ -s out.txt ]] && exit<br />
成功。<br />
其实当文件存在的情况下,shell的条件判断还有很多很多种可能,列表如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>-b file 若文件存在且是一个块特殊文件,则为真
-c file 若文件存在且是一个字符特殊文件,则为真
-d file 若文件存在且是一个目录,则为真
-e file 若文件存在,则为真
-f file 若文件存在且是一个规则文件,则为真
-g file 若文件存在且设置了SGID位的值,则为真
-h file 若文件存在且为一个符合链接,则为真
-k file 若文件存在且设置了"sticky"位的值
-p file 若文件存在且为一已命名管道,则为真
-r file 若文件存在且可读,则为真
-s file 若文件存在且其大小大于零,则为真
-u file 若文件存在且设置了SUID位,则为真
-w file 若文件存在且可写,则为真
-x file 若文件存在且可执行,则为真
-o file 若文件存在且被有效用户ID所拥有,则为真
</code></pre>
</div>
squid的snmp配置和部分oid说明
2010-09-14T00:00:00+08:00
monitor
squid
snmp
http://chenlinux.com/2010/09/14/intro-snmp-config-and-oids-of-squid
<p>首先配置squid.conf如下:<br />
<code class="highlighter-rouge">squid
acl MonitorCenter src 127.0.0.1
acl snmppublic snmp_community public
snmp_access allow snmppublic MonitorCenter
snmp_access deny all
</code></p>
<p>然后配置snmpd.conf如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>com2sec notConfigUser default public
group notConfigGroup v2c notConfigUser
view systemview included .1.3.6.1.4.1.3495.1
proxy -v 1 -c public 127.0.0.1:3401 .1.3.6.1.4.1.3495.1
</code></pre>
</div>
<p>分别重启squid和snmpd即可。<br />
用snmpwalk -v 2c -c public 192.168.0.2 1.3.6.1.4.1.3495 -Cc就可以查看全部的数据了。对了解前段缓存有用的oid主要如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>SNMPv2-SMI::enterprises.3495.1.1.1.0 = INTEGER: 286800 内存缓存文件大小KB
SNMPv2-SMI::enterprises.3495.1.1.2.0 = INTEGER: 1072480 磁盘缓存文件大小KB
SNMPv2-SMI::enterprises.3495.1.3.1.3.0 = INTEGER: 387200 内存使用大小KB
SNMPv2-SMI::enterprises.3495.1.3.1.5.0 = INTEGER: 8 进程使用CPU的比例
SNMPv2-SMI::enterprises.3495.1.3.1.7.0 = Gauge32: 66757 缓存文件总数
SNMPv2-SMI::enterprises.3495.1.3.1.10.0 = Gauge32: 225 可用文件描述符个数
SNMPv2-SMI::enterprises.3495.1.3.1.12.0 = Gauge32: 799 当前使用中的文件描述符个数
SNMPv2-SMI::enterprises.3495.1.3.2.1.15.0 = Gauge32: 90377 当前访问缓存的客户端总数
SNMPv2-SMI::enterprises.3495.1.3.2.2.1.9.1 = INTEGER: 89 一分钟请求命中比
SNMPv2-SMI::enterprises.3495.1.3.2.2.1.10.1 = INTEGER: 94 一分钟字节命中比
SNMPv2-SMI::enterprises.3495.1.3.2.2.1.3.1 = INTEGER: 7 一分钟TCP_MISS/200比
SNMPv2-SMI::enterprises.3495.1.3.2.2.1.4.1 = INTEGER: 0 一分钟TCP_IMS_HIT/304比
SNMPv2-SMI::enterprises.3495.1.3.2.2.1.5.1 = INTEGER: 0 一分钟TCP_HIT/200比
SNMPv2-SMI::enterprises.3495.1.3.2.2.1.11.1 = INTEGER: 45 一分钟TCP_REFRESH_HIT/304比
++++++++以上oid最后一位都可以改成任意时间(分钟为单位)进行统计++++++++
SNMPv2-SMI::enterprises.3495.1.5.1.1.9.10.10.10.13 = Counter32: 324396 请求回某源的总数
SNMPv2-SMI::enterprises.3495.1.5.2.1.2.58.31.229.71 = Counter32: 19 某客户ip的http请求数
SNMPv2-SMI::enterprises.3495.1.5.2.1.3.58.31.229.71 = Counter32: 327 响应该IP请求的字节数KB
SNMPv2-SMI::enterprises.3495.1.5.2.1.4.58.31.229.71 = Counter32: 17 响应该IP请求的命中率
SNMPv2-SMI::enterprises.3495.1.5.2.1.5.58.31.229.71 = Counter32: 324 响应该IP请求的字节命中数KB
</code></pre>
</div>
snmpwalk的报错一例
2010-09-14T00:00:00+08:00
monitor
snmp
http://chenlinux.com/2010/09/14/an-oid-error-of-snmpwalk
<p>在给squid配置了snmp后,用snmpwalk采集数据,总览一下:</p>
<p>snmpwalk 192.168.0.2 -v 2c -c public 1.3.6.1.4.1.3495</p>
<p>刷到最后爆出错误:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>SNMPv2-SMI::enterprises.3495.1.5.2.1.1.218.28.141.150 = IpAddress: 218.28.141.150
SNMPv2-SMI::enterprises.3495.1.5.2.1.1.119.184.18.137 = IpAddress: 119.184.18.137
Error: OID not increasing: SNMPv2-SMI::enterprises.3495.1.5.2.1.1.218.28.141.150
>= SNMPv2-SMI::enterprises.3495.1.5.2.1.1.119.184.18.137
</code></pre>
</div>
<p>原来snmp默认在抓取oid的时候,是按顺序递增下去请求的,而squid的最后一个1.5.2.1类别,是客户端数据,IP是分散的,所以218没法增长到119,于是就出错退出了。<br />
解决方法很简单,加上-Cc参数即可,help说明如下:</p>
<p>-C APPOPTS Set various application specific behaviours:<br />
p: print the number of variables found<br />
i: include given OID in the search range<br />
I: don’t include the given OID, even if no results are returned<br />
c: do not check returned OIDs are increasing<br />
t: Display wall-clock time to complete the request</p>
<p>不过这里加上-Cc后,简直就是在刷屏了,无数客户端ip全部都要打印出来,慎用慎用……</p>
BSD下的字符串运算
2010-08-25T00:00:00+08:00
bash
FreeBSD
http://chenlinux.com/2010/08/25/string-operator-on-bsd
<p>之前在linux上有个脚本,通过expr命令截取字符串的。大意如下:<br />
a=/path/to/example<br />
b=<code class="highlighter-rouge">expr length "$a "</code><br />
c=/path/to/example/file/to/example<br />
d=<code class="highlighter-rouge">expr length "$c"</code><br />
e=<code class="highlighter-rouge">expr substr "$c" "$b" "$d"</code><br />
转移到BSD后,脚本报错:expr: syntax error<br />
分别在linux和bsd上man expr后对比了一下,发现bsd上的expr确实没有index、length、substr等运算,原来linux上的expr是GNU的;而bsd上的expr是POSIX的,没有gnu的那些扩展用法……<br />
于是必须使用些通用的办法来完成这个截取功能。方法很多,举例如下:</p>
<p>1、awk法<br />
awk ‘BEGIN{print length(‘$a’)}’;<br />
awk ‘BEGIN{print substr(‘$c’,’$b’,’$d’)}’</p>
<p>2、bash扩展法<br />
${#a}<br />
${c:$b:${#c}}</p>
<p>3、标准expr+cut法<br />
expr “$a “ : “.*”<br />
echo $c | cut -c $b-$d</p>
<p>POSIX下的expr没有length,不过man中提供了采用:匹配.*的方法获取长度;</p>
<p>GNU下的expr substr和awk的substr函数一样,都是从$b位开始,截取$d位数的字符子串;而cut命令则是从$b位开始截取到$d位为止的字符子串。</p>
<p>其实cut这种方法才是最符合前面的length的,不过substr的时候,位数多设一些,也不影响结果~~</p>
收到jwplayer的推广邮件
2010-08-24T00:00:00+08:00
http://chenlinux.com/2010/08/24/received-a-mail-from-jwplayer
<p>在做CDN的时候,曾经有一个客户加速视频播放,所以去注册下载了JWPlayer尝尝鲜。<br />
转眼几个月了,今天突然收到一封信,《Introducing Open Video Ads》,原来是JWPlayer的开发者,新出的基于GPL3的一个开源插件,可以在jwplayer和flowplayer播放正式视频前,插播一段自定义广告~~<br />
记得离职前还有同事专门在研究如何在播放rtmp流时切换插播广告,不知道现在搞定没。<br />
不过,这还是我第一次收到开源软件开发者推广自己软件的邮件。。。</p>
pmap命令
2010-08-24T00:00:00+08:00
linux
http://chenlinux.com/2010/08/24/intro-pmap
<p>继pgrep之后,又发现一个pmap命令,有些不错的小作用~<br />
比方说,用pgrep java得出pid后,用pmap $pid,得出输出结果如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>23792: /usr/java/jdk1.5.0_14/bin/java -Djava.util.logging.manager=com.caucho.log.LogManagerImpl -Djava.system.class.loader=com.caucho.loader.SystemClassLoader -Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl -Djava.awt.headless=true -Dresin.home=/usr/local/resin3.1.8-rtuku/ -Xmx256m -Xss1m -Xdebug -Dcom.sun.management.jmxremote -Djava.util.logging.manager=com.caucho.log.LogManagerImpl -Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl -Djava.awt.headless=true -Dresin.
0028f000 72K r-x-- /lib/libnsl-2.3.4.so
002a1000 8K rwx-- /lib/libnsl-2.3.4.so
002a3000 8K rwx-- [ anon ]
0031c000 84K r-x-- /lib/ld-2.3.4.so
00331000 4K r-x-- /lib/ld-2.3.4.so
00332000 4K rwx-- /lib/ld-2.3.4.so
0033a000 1172K r-x-- /lib/tls/libc-2.3.4.so
0045f000 4K r-x-- /lib/tls/libc-2.3.4.so
00460000 12K rwx-- /lib/tls/libc-2.3.4.so
00463000 8K rwx-- [ anon ]
00467000 132K r-x-- /lib/tls/libm-2.3.4.so
……
b7f50000 4K rwx-- [ anon ]
b7f51000 4K r-x-- [ anon ]
b7f52000 4K r-x-- [ anon ]
bfcf1000 12K ----- [ anon ]
bfcf4000 1012K rwx-- [ stack ]
total 652340K
</code></pre>
</div>
<p>从中可以看出来加载的所有so和线程堆栈用掉的内存。据称,当anon在512K-4M之间的超过上千个的时候,可能就是在多线程上有问题了。</p>
<p>还有一个用途,比较偏门的。<br />
比如一个squid服务器,前人直接cd进目录,然后./squid启用的服务。那怎么去知道这个squid到底在那个目录里呢?(尤其是发现惯用的/usr/local下哗哗的摆着五个squid目录……)</p>
<p>现在只要pmap 一下,第一条就给出了全路径。哈哈~~</p>
inotify+purge_cache
2010-08-24T00:00:00+08:00
linux
inotify
bash
http://chenlinux.com/2010/08/24/inotifypurge_cache
<p>linux内核从2.6.13开始,加入了inotify特性。对目录、文件的各种修改,都会发出inotify信号。包括<br />
IN_ACCESS<br />
IN_MODIFY<br />
IN_ATTRIB<br />
IN_CLOSE_WRITE<br />
IN_CLOSE_NOWRITE<br />
IN_OPEN<br />
IN_MOVED_FROM<br />
IN_MOVED_TO<br />
IN_CREATE<br />
IN_DELETE<br />
IN_DELETE_SELF<br />
IN_MOVE_SELF<br />
IN_UNMOUNT<br />
IN_CLOSE<br />
IN_MOVE<br />
目前最常见的inotify应用,就是和rsync配合进行实时同步。<br />
而对web发布路径进行inotify监听的话,可以实时PURGE掉前端cache,保证网民访问的实效性。内容更新周期不固定的一些网站,大可以设定长一些的expires(也不要太长,不然浏览器端本身的缓存影响比较大),然后通过inotify监听来强制控制缓存时间,应该是比较有效果的。<br />
最早的思路,先用perl的Linux::Inotify模块watch目录,read出每次event的name;再用IO::Socket模块向squid发送”PURGE $url HTTP/1.0\n\n”请求,最后用WWW::Curl::Form模块POST数据到CDN的刷新接口。洋洋洒洒好几十行后,发现利用inotify-tools、curl、squidclient等现成的工具,写成的shell脚本更加简单而且方便。<br />
先修改squid.conf,添加web服务器ip的purge权限,重读配置;<br />
在web服务器上,从sourceforge下载inotify-tools源码编译:wget http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz && tar zxf inotify-tools-3.14.tar.gz && cd inotify-tools-3.14 && ./configure –prefix=/usr && make && make install<br />
从squid上scp /usr/local/squid/bin/squidclient 到web服务器上;<br />
要是没有curl的话,yum install一个。<br />
最后创建inotify-purge.sh脚本如下:<br />
```bash<br />
#!/bin/bash<br />
WEB_DIR=/path/to/example<br />
IPLIST=”1.2.3.4<br />
1.2.3.5<br />
1.2.3.6<br />
1.2.3.7<br />
1.2.3.8<br />
“</p>
<p>inotifywait -mrq –format ‘%f’ -e modify,delete,create,move ${WEB_DIR} | while read file<br />
do<br />
PURGE_URL=<code class="highlighter-rouge">echo "http://dvs.china.com/$file"</code><br />
for i in $IPLIST;do<br />
/home/tools/squidclient -p 80 -h $i -m purge “$PURGE_URL”<br />
done<br />
curl -s -d “username=test&password=123456&type=1&url=$PURGE_URL” http://pushwt.dnion.com/cdnUrlPush.do<br />
done<br />
```</p>
nagios绘图
2010-08-13T00:00:00+08:00
monitor
nagios
rrdtools
http://chenlinux.com/2010/08/13/intro-pnp4nagios
<p>nagios默认command中,有个未开启的process_performance_data。可以开启它来保存数据,然后提供给rrdtools绘图。</p>
<p>下载pnp插件包,官网是<a href="http://www.pnp4nagios.org/">http://www.pnp4nagios.org</a>,和cacti一样,保证有lamp、rrdtool(低版本的还要GD),然后和其他插件一样./configure && make && make all && make install && make install-config && make install-init就行了。<br />
然后修改nagios.cfg如下:</p>
<p>process_performance_data=1<br />
service_perfdata_command=process-service-perfdata<br />
host_perfdata_file=/usr/local/nagios/var/host-perfdata</p>
<p>修改commands.cfg如下:</p>
<p>define command{<br />
command_name process-service-perfdata-file<br />
command_line $USER1$/process_perfdata.pl<br />
}<br />
define command{<br />
command_name process-host-perfdata-file<br />
command_line $USER1$/process_perfdata.pl<br />
}</p>
<p>再修改pnp/process_perfdata.cfg-sample,设定好rrdtool等的路径,另存为process_perfdata.cfg。重启nagios即可。</p>
<p>过一会图就出来了,不过我自己写的两个脚本,随意检测正常,页面上也看到了输出值,就是没图……报出来***.rrd not found。进目录一看,其他图都创建出来了,就自定义脚本的没有……</p>
<p>咬牙看了会include/function.inc.php,发现php在调用rrdtool create的时候,使用的直接就是$data[1][‘*‘]这样的数据,整个functions里都没有看到怎么匹配切割数据的部分。可是看页面上nagios自带的command输出,也没什么明显标记啊?</p>
<table>
<tbody>
<tr>
<td>上服务器看脚本,试着手动运行了一次check_http,发现输出结果比页面上显示的还多了一串“</td>
<td>time=0.003329s;5.000000;10.000000;0.000000 size=1576B;;;0”!这个明显有着固定的格式!</td>
</tr>
</tbody>
</table>
<p>然后在页面上点开check_http看,发现在报警状态“Status Information: HTTP OK HTTP/1.0 200 OK - 1576 bytes in 0.003 seconds”下还有一行“ Performance Data: time=0.003329s;5.000000;10.000000;0.000000 size=1576B;;;0”。原来如此……</p>
<table>
<tbody>
<tr>
<td>修改shell脚本的echo内容,也照葫芦画瓢的加上了“</td>
<td>port ${PORT}=${PORT_CONN};${WARNING};${CRITICAL};;”等五分钟后,再点开pnp图,果然出现了!</td>
</tr>
</tbody>
</table>
gnuplot画图
2010-08-13T00:00:00+08:00
monitor
gnuplot
http://chenlinux.com/2010/08/13/intro-gnuplot
<p>之前提高日志流量统计的问题。并给出了分时流量的计算方法。当是想的是用perl调用rrd或者gd画图,甚至有把日志统计也写成perl的打算。<br />
不过今天发现了一个小工具gnuplot,画图功能相当强大(在找资料的时候看到台湾有人用这个画台湾的三维地形图!),上手相当简单,尤其适合日志统计分析,相比来说,rrd还是比较适合实时监控画图的情况。<br />
比如当初的脚本输出如下:<br />
[root@Zabbix cache]# tail test.log<br />
23:14 506.877<br />
23:19 501.068<br />
23:24 493.254<br />
23:29 469.184<br />
23:34 460.161<br />
23:39 426.065<br />
23:44 429.734<br />
23:49 409.255<br />
23:54 423.512<br />
23:59 390.676<br />
然后编写gnuplot的配置文件如下:<br />
<code class="highlighter-rouge">bash
[root@Zabbix cache]# cat log.conf
set terminal png truecolor size 550,250 #指定输出成png图片,且图片大小为550×250,需要ligpng支持,采用默认颜色设定
set output "log.png" #指定输出png图片的文件名
set autoscale #轴向标记自动控制
set xdata time #X轴数据格式为时间
set timefmt "%H:%M" #时间输入格式为"小时:分钟"
set style data lines #数据显示方式为连线
set xlabel "time per day" #X轴标题
set ylabel "Mbps" #Y轴标题
set title "image.tuku.china.com flow" #图片标题
set grid #显示网格
plot "test.log" using 1:2 title "access_flow" #从test.log文件中读取第一列和第二列作为X轴和Y轴数据,示例名"log_flow"
</code><br />
最后运行cat log.conf | gnuplot命令,就生成了log.png文件,如下:<br />
<img src="/images/uploads/gnuplot.png" alt="" /><br />
就是不知道X轴上这个01/01怎么消除掉……<br />
对比帝联提供的flash图片如下:<br />
<img src="/images/uploads/dilian.jpg" alt="" /><br />
可以看出基本是一致的。</p>
bc命令及其他
2010-08-13T00:00:00+08:00
bash
http://chenlinux.com/2010/08/13/intro-bc-command
<p>在写check_if_flow.sh的时候,因为要比较的值太多,几个网卡,每个都分进出、然后分w和c。如果直接if判断大小,会写的无比庞大……于是想到根据比较结果先输出一个变量,大就是1,小就是0,类似这种。</p>
<p>于是找到了bc命令,最终结果如下:</p>
<p>[root@test ~]# echo “1.13 > 1.2”|bc<br />
0<br />
[root@test ~]# echo “1.13 < 1.2”|bc<br />
1<br />
bc支持+-*/%^=!><各种运算,还能通过scale指定小数点后几位;<br />
还能任意转换数字的进制,如下:</p>
<p>[root@test ~]# echo ‘obase=10; ibase=16; 1E79’ | bc<br />
7801</p>
<p>相比之前脚本里用的awk要灵活(就是必须大写)。awk从十进制变十六进制很容易,变回去就难了……得用上函数才行:</p>
<p>[root@test ~]# echo ‘1E79’|awk ‘{printf “%s”,strtonum(“0x”$1)}’<br />
7801</p>
<p>而shell相反,从十六进制变十进制很容易,反过来去难……</p>
<p>[root@test ~]# echo $[16#1e79]<br />
7801</p>
<p>ksh中可以指定typeset -i,bash没找到~所以只能把别的进制改成10进制</p>
<p>[root@test ~]# ksh<br />
# typeset -i16 num=7801<br />
# echo $num<br />
16#1e79</p>
流量监控
2010-08-05T00:00:00+08:00
monitor
perl
nagios
http://chenlinux.com/2010/08/05/traffic-monitor
<p>流量监控,一般看cacti上的绘图。近日打算设置报警,懒得给cacti加模块,自己写个脚本吧,于是开始研究这个流量监控的方式。<br />
先是在网上看到一个nagios的check_traffic.sh脚本,核心就是用snmpwalk取网卡总流量,写在/tmp/某个文件下,下次nrpe启动check时,再去新的总流量,减去文件中读取出来的值,除以启动间隔时间,就是平均流量值。<br />
用snmpwalk -v 2c -c public localhost IF-MIB::ifInOctets取出值来一看,发现和ifconifg出来的RX数值是一样的!<br />
然后有张宴的net.sh脚本,从/proc/net/dev中取值,然后存进变量后,sleep一定时间,再取一次,同样相减再做除法,得出平均流量值。<br />
再cat /proc/net/dev和ifconfig的一比较,数值也是一样的,把两个脚本设定相同间隔,同时运行,显示的结果都是一样的!<br />
那从本机监控的角度来说,那当然是从proc中取值计算容易了。毕竟给一大把机器装snmpwalk很费的……<br />
(因为近期cacti上的图时不时有某些机器突发满载流量尖峰,持续时间又很短,所以靠cacti或者nagios本身这种间隔性轮询扫描很可能就错过去了)<br />
最终想法是,在机器上后台长期运行监控脚本,碰到流量突发,发送到监控服务器,监控服务器上开启sniffer或者wireshark抓包,同时发邮件报警。<br />
目前初步完成监控客户端脚本如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/perl</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">IO::</span><span class="nv">Socket</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Getopt::</span><span class="nv">Long</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">POSIX</span> <span class="sx">qw(strftime)</span><span class="p">;</span>
<span class="c1">#Init</span>
<span class="k">my</span> <span class="nv">%traf</span><span class="p">;</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$warning</span><span class="p">,</span><span class="nv">$interval</span><span class="p">,</span><span class="nv">$usage</span><span class="p">,</span><span class="nv">$silent</span><span class="p">);</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$eth0_in</span><span class="p">,</span><span class="nv">$eth0_out</span><span class="p">,</span><span class="nv">$eth1_in</span><span class="p">,</span><span class="nv">$eth1_out</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$alarm</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="nv">$warning</span> <span class="o">=</span> <span class="s">"1000,2000,10000,80000"</span><span class="p">;</span>
<span class="nv">$interval</span> <span class="o">=</span> <span class="mi">10</span><span class="p">;</span>
<span class="c1">#get options</span>
<span class="nn">Getopt::Long::</span><span class="nv">Configure</span><span class="p">(</span><span class="s">'bundling'</span><span class="p">);</span>
<span class="nv">GetOptions</span><span class="p">(</span>
<span class="s">"h"</span> <span class="o">=></span> <span class="nv">$usage</span><span class="p">,</span> <span class="s">"v"</span> <span class="o">=></span> <span class="nv">$usage</span><span class="p">,</span>
<span class="s">"s"</span> <span class="o">=</span> <span class="nv">$silent</span><span class="p">,</span>
<span class="s">"w=s"</span> <span class="o">=></span> <span class="nv">$warning</span><span class="p">,</span>
<span class="s">"H=s"</span> <span class="o">=></span> <span class="nv">$peer</span><span class="p">,</span>
<span class="s">"t=f"</span> <span class="o">=></span> <span class="nv">$interval</span><span class="p">,</span>
<span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$usage</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">usage</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$warning</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">usage</span> <span class="k">unless</span> <span class="nv">$warning</span> <span class="o">=~</span> <span class="sr">/d+,d+,d+,d+/</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$eth0_in_warn</span><span class="p">,</span><span class="nv">$eth0_out_warn</span><span class="p">,</span><span class="nv">$eth1_in_warn</span><span class="p">,</span><span class="nv">$eth1_out_warn</span><span class="p">)</span> <span class="o">=</span> <span class="nb">split</span><span class="p">(</span><span class="sr">/,/</span><span class="p">,</span><span class="nv">$warning</span><span class="p">);</span>
<span class="c1">#fork</span>
<span class="nb">defined</span><span class="p">(</span><span class="k">my</span> <span class="nv">$pid</span><span class="o">=</span><span class="nb">fork</span><span class="p">)</span> <span class="ow">or</span> <span class="nb">die</span> <span class="s">"Cant fork:$!"</span><span class="p">;</span>
<span class="k">unless</span><span class="p">(</span><span class="nv">$pid</span><span class="p">){</span>
<span class="p">}</span><span class="k">else</span><span class="p">{</span>
<span class="nb">exit</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">#main</span>
<span class="k">while</span> <span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">#这点很奇怪,本打算把main里头的东西直接写在while里的,运行却一直没输出;只要定义成sub,立马就好用……</span>
<span class="nv">main</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">#functions</span>
<span class="k">sub </span><span class="nf">main</span> <span class="p">{</span>
<span class="p">(</span><span class="nv">$eth0_in</span><span class="p">,</span><span class="nv">$eth0_out</span><span class="p">,</span><span class="nv">$eth1_in</span><span class="p">,</span><span class="nv">$eth1_out</span><span class="p">)</span> <span class="o">=</span> <span class="nv">count</span><span class="p">;</span>
<span class="c1">#每5分钟保存到tmp文件,供nrpe读取数据,给nagios画图用</span>
<span class="k">my</span> <span class="nv">$nrpetime</span> <span class="o">=</span> <span class="nv">strftime</span><span class="p">(</span><span class="s">"%M%S"</span><span class="p">,</span><span class="nb">localtime</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$nrpetime</span> <span class="o">=~</span> <span class="sr">/d(5|0)0d/</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">write_for_nrpe</span><span class="p">;</span>
<span class="p">}</span>
<span class="nb">alarm</span> <span class="k">unless</span><span class="p">(</span><span class="nv">$silent</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">count</span> <span class="p">{</span>
<span class="nv">data</span><span class="p">;</span>
<span class="nv">$traf</span><span class="p">{</span><span class="s">"eth0In_old"</span><span class="p">}</span> <span class="o">=</span> <span class="nv">$traf</span><span class="p">{</span><span class="nv">eth0In</span><span class="p">};</span>
<span class="nv">$traf</span><span class="p">{</span><span class="s">"eth1In_old"</span><span class="p">}</span> <span class="o">=</span> <span class="nv">$traf</span><span class="p">{</span><span class="nv">eth1In</span><span class="p">};</span>
<span class="nv">$traf</span><span class="p">{</span><span class="s">"eth0Out_old"</span><span class="p">}</span> <span class="o">=</span> <span class="nv">$traf</span><span class="p">{</span><span class="nv">eth0Out</span><span class="p">};</span>
<span class="nv">$traf</span><span class="p">{</span><span class="s">"eth1Out_old"</span><span class="p">}</span> <span class="o">=</span> <span class="nv">$traf</span><span class="p">{</span><span class="nv">eth1Out</span><span class="p">};</span>
<span class="nb">sleep</span> <span class="nv">$interval</span><span class="p">;</span>
<span class="nv">data</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$eth0_in_flow</span> <span class="o">=</span> <span class="nb">sprintf</span> <span class="s">"%.2f"</span><span class="p">,(</span><span class="nv">$traf</span><span class="p">{</span><span class="nv">eth0In</span><span class="p">}</span><span class="o">-</span><span class="nv">$traf</span><span class="p">{</span><span class="s">"eth0In_old"</span><span class="p">})</span><span class="sr">/$interval*8/</span><span class="mi">1024</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$eth1_in_flow</span> <span class="o">=</span> <span class="nb">sprintf</span> <span class="s">"%.2f"</span><span class="p">,(</span><span class="nv">$traf</span><span class="p">{</span><span class="nv">eth1In</span><span class="p">}</span><span class="o">-</span><span class="nv">$traf</span><span class="p">{</span><span class="s">"eth1In_old"</span><span class="p">})</span><span class="sr">/$interval*8/</span><span class="mi">1024</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$eth0_out_flow</span> <span class="o">=</span> <span class="nb">sprintf</span> <span class="s">"%.2f"</span><span class="p">,(</span><span class="nv">$traf</span><span class="p">{</span><span class="nv">eth0Out</span><span class="p">}</span><span class="o">-</span><span class="nv">$traf</span><span class="p">{</span><span class="s">"eth0Out_old"</span><span class="p">})</span><span class="sr">/$interval*8/</span><span class="mi">1024</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$eth1_out_flow</span> <span class="o">=</span> <span class="nb">sprintf</span> <span class="s">"%.2f"</span><span class="p">,(</span><span class="nv">$traf</span><span class="p">{</span><span class="nv">eth1Out</span><span class="p">}</span><span class="o">-</span><span class="nv">$traf</span><span class="p">{</span><span class="s">"eth1Out_old"</span><span class="p">})</span><span class="sr">/$interval*8/</span><span class="mi">1024</span><span class="p">;</span>
<span class="k">return</span> <span class="nv">$eth0_in_flow</span><span class="p">,</span><span class="nv">$eth0_out_flow</span><span class="p">,</span><span class="nv">$eth1_in_flow</span><span class="p">,</span><span class="nv">$eth1_out_flow</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">data</span> <span class="p">{</span>
<span class="nb">open</span> <span class="nv">DEV</span><span class="p">,</span><span class="s">"&lt;/proc/net/dev"</span> <span class="o">||</span> <span class="nb">die</span> <span class="s">"Cannot open procfs!"</span><span class="p">;</span>
<span class="k">while</span> <span class="p">(</span><span class="nb">defined</span><span class="p">(</span><span class="k">my</span> <span class="nv">$ifdata</span><span class="o">=&</span><span class="ow">lt</span><span class="p">;</span><span class="nv">DEV</span><span class="o">></span><span class="p">)){</span>
<span class="k">next</span> <span class="k">if</span> <span class="nv">$ifdata</span> <span class="o">!~</span> <span class="sr">/eth/</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@data</span> <span class="o">=</span> <span class="nb">split</span> <span class="p">(</span><span class="sr">/:|s+/</span><span class="p">,</span><span class="nv">$ifdata</span><span class="p">);</span>
<span class="nv">$traf</span><span class="p">{</span><span class="s">"$data[1]In"</span><span class="p">}</span> <span class="o">=</span> <span class="nv">$data</span><span class="p">[</span><span class="mi">2</span><span class="p">];</span>
<span class="nv">$traf</span><span class="p">{</span><span class="s">"$data[1]Out"</span><span class="p">}</span> <span class="o">=</span> <span class="nv">$data</span><span class="p">[</span><span class="mi">10</span><span class="p">];</span>
<span class="p">}</span>
<span class="nb">close</span> <span class="nv">DEV</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">write_for_nrpe</span> <span class="p">{</span>
<span class="nb">open</span> <span class="nv">FH</span><span class="p">,</span><span class="s">">/tmp/if_flow.txt"</span> <span class="o">||</span> <span class="nb">die</span> <span class="vg">$!</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">FH</span> <span class="s">"$eth0_in|$eth0_out|$eth1_in|$eth1_out"</span><span class="p">;</span>
<span class="nb">close</span> <span class="nv">FH</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">alarm</span> <span class="p">{</span>
<span class="c1">#考虑到默认10s取值一次,突发流量如果持续100s,就会向socket发送10次,所以进行判定,只在突发开始和突发结束时发送warn和ok。写的很初级……</span>
<span class="k">my</span> <span class="nv">$alarm_int</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="nv">$alarm_int</span> <span class="o">=</span> <span class="mi">1</span> <span class="k">if</span> <span class="p">(</span><span class="nv">$eth0_in</span><span class="o">-</span><span class="nv">$eth0_in_warn</span><span class="o">></span><span class="mi">0</span><span class="p">);</span>
<span class="nv">$alarm_int</span> <span class="o">=</span> <span class="mi">1</span> <span class="k">if</span> <span class="p">(</span><span class="nv">$eth0_out</span><span class="o">-</span><span class="nv">$eth0_out_warn</span><span class="o">></span><span class="mi">0</span><span class="p">);</span>
<span class="nv">$alarm_int</span> <span class="o">=</span> <span class="mi">2</span> <span class="k">if</span> <span class="p">(</span><span class="nv">$eth1_in</span><span class="o">-</span><span class="nv">$eth1_in_warn</span><span class="o">></span><span class="mi">0</span><span class="p">);</span>
<span class="nv">$alarm_int</span> <span class="o">=</span> <span class="mi">2</span> <span class="k">if</span> <span class="p">(</span><span class="nv">$eth1_out</span><span class="o">-</span><span class="nv">$eth1_out_warn</span><span class="o">></span><span class="mi">0</span><span class="p">);</span>
<span class="k">next</span> <span class="k">if</span> <span class="nv">$alarm_int</span> <span class="o">==</span> <span class="nv">$alarm</span><span class="p">;</span>
<span class="nv">call_sniffer</span><span class="p">(</span><span class="s">"eth0:WARN"</span><span class="p">)</span> <span class="k">if</span> <span class="p">(</span><span class="nv">$alarm_int</span><span class="o">-</span><span class="nv">$alarm</span><span class="o">==</span><span class="mi">1</span><span class="p">);</span>
<span class="nv">call_sniffer</span><span class="p">(</span><span class="s">"eth1:WARN"</span><span class="p">)</span> <span class="k">if</span> <span class="p">(</span><span class="nv">$alarm_int</span><span class="o">-</span><span class="nv">$alarm</span><span class="o">==</span><span class="mi">2</span><span class="p">);</span>
<span class="nv">call_sniffer</span><span class="p">(</span><span class="s">"eth0:OK"</span><span class="p">)</span> <span class="k">if</span> <span class="p">(</span><span class="nv">$alarm_int</span><span class="o">-</span><span class="nv">$alarm</span><span class="o">==-</span><span class="mi">1</span><span class="p">);</span>
<span class="nv">call_sniffer</span><span class="p">(</span><span class="s">"eth1:OK"</span><span class="p">)</span> <span class="k">if</span> <span class="p">(</span><span class="nv">$alarm_int</span><span class="o">-</span><span class="nv">$alarm</span><span class="o">==-</span><span class="mi">2</span><span class="p">);</span>
<span class="nv">$alarm</span> <span class="o">=</span> <span class="nv">$alarm_int</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">call_sniffer</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$message</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$socket</span> <span class="o">=</span> <span class="nn">IO::Socket::</span><span class="nv">INET</span><span class="o">-></span><span class="k">new</span><span class="p">(</span><span class="nv">PeerAddr</span> <span class="o">=></span> <span class="nv">$peer</span><span class="p">,</span>
<span class="nv">PeerPort</span> <span class="o">=></span> <span class="mi">12345</span><span class="p">,</span>
<span class="nv">Proto</span><span class="err"> </span> <span class="o">=></span> <span class="s">'tcp'</span><span class="p">)</span>
<span class="ow">or</span> <span class="nb">die</span> <span class="vg">$@</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">$socket</span> <span class="s">"${message}n"</span><span class="p">;</span>
<span class="nv">$socket</span><span class="o">-></span><span class="nb">shutdown</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$answer</span> <span class="o">=</span> <span class="sr"><$socket></span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$answer</span><span class="p">)</span> <span class="p">{</span>
<span class="k">print</span> <span class="nv">$answer</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">$socket</span><span class="o">-></span><span class="nb">close</span> <span class="ow">or</span> <span class="nb">die</span> <span class="vg">$!</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">usage</span> <span class="p">{</span>
<span class="k">print</span> <span class="s">"Version: check_eth_flow.pl v0.1n"</span><span class="p">;</span>
<span class="k">print</span> <span class="s">"Usage: check_eth_flow.pl -w 1000,2000,10000,80000 -t 10n"</span><span class="p">;</span>
<span class="k">print</span> <span class="s">"tt-w Warning Value: eth0_in,eth0_out,eth1_in,eth1_out;n"</span><span class="p">;</span>
<span class="k">print</span> <span class="s">"tt-t Interval Time;n"</span><span class="p">;</span>
<span class="k">print</span> <span class="s">"tt-s Silent write for nagios;n"</span>
<span class="k">print</span> <span class="s">"tt-H Host address of peer;n"</span><span class="p">;</span>
<span class="k">print</span> <span class="s">"tt-h Print this usage.n"</span><span class="p">;</span>
<span class="nb">exit</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
<p>nrpe的check_if_flow.sh就比较简单了,如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="k">while </span><span class="nb">getopts</span> <span class="s2">"w:c:h"</span> OPT;do
<span class="k">case</span> <span class="nv">$OPT</span> <span class="k">in
</span>w<span class="p">)</span>
<span class="nv">WARNING</span><span class="o">=</span><span class="k">${</span><span class="nv">OPTARG</span><span class="k">}</span>
<span class="nv">eth0_in_warn</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="nv">$WARNING</span>|awk -F, <span class="s1">'{print $1}'</span><span class="sb">`</span>
<span class="nv">eth0_out_warn</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="nv">$WARNING</span>|awk -F, <span class="s1">'{print $2}'</span><span class="sb">`</span>
<span class="nv">eth1_in_warn</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="nv">$WARNING</span>|awk -F, <span class="s1">'{print $3}'</span><span class="sb">`</span>
<span class="nv">eth1_out_warn</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="nv">$WARNING</span>|awk -F, <span class="s1">'{print $4}'</span><span class="sb">`</span>
<span class="p">;;</span>
c<span class="p">)</span>
<span class="nv">CRITICAL</span><span class="o">=</span><span class="k">${</span><span class="nv">OPTARG</span><span class="k">}</span>
<span class="nv">eth0_in_critical</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="nv">$CRITICAL</span>|awk -F, <span class="s1">'{print $1}'</span><span class="sb">`</span>
<span class="nv">eth0_out_critical</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="nv">$CRITICAL</span>|awk -F, <span class="s1">'{print $2}'</span><span class="sb">`</span>
<span class="nv">eth1_in_critical</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="nv">$CRITICAL</span>|awk -F, <span class="s1">'{print $3}'</span><span class="sb">`</span>
<span class="nv">eth1_out_critical</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="nv">$CRITICAL</span>|awk -F, <span class="s1">'{print $4}'</span><span class="sb">`</span>
<span class="p">;;</span>
<span class="k">*</span><span class="p">)</span>
<span class="nb">echo</span> <span class="s2">"Usage: </span><span class="nv">$0</span><span class="s2"> -w 500,500,1000,1000 -c 2000,2000,10000,10000"</span>
<span class="nb">exit </span>0
<span class="p">;;</span>
<span class="k">esac</span>
<span class="k">done
</span><span class="nv">eth0_in</span><span class="o">=</span><span class="sb">`</span>cat /tmp/if_flow.txt|awk -F| <span class="s1">'{print $1}'</span><span class="sb">`</span>
<span class="nv">eth0_out</span><span class="o">=</span><span class="sb">`</span>cat /tmp/if_flow.txt|awk -F| <span class="s1">'{print $2}'</span><span class="sb">`</span>
<span class="nv">eth1_in</span><span class="o">=</span><span class="sb">`</span>cat /tmp/if_flow.txt|awk -F| <span class="s1">'{print $3}'</span><span class="sb">`</span>
<span class="nv">eth1_out</span><span class="o">=</span><span class="sb">`</span>cat /tmp/if_flow.txt|awk -F| <span class="s1">'{print $4}'</span><span class="sb">`</span>
<span class="nv">eth0_in_diff</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$eth0_in</span><span class="s2"> &lt; </span><span class="nv">$eth0_in_warn</span><span class="s2">"</span>|bc<span class="sb">`</span>
<span class="nv">eth0_out_diff</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$eth0_out</span><span class="s2"> &lt; </span><span class="nv">$eth0_out_warn</span><span class="s2">"</span>|bc<span class="sb">`</span>
<span class="nv">eth1_in_diff</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$eth1_in</span><span class="s2"> &lt; </span><span class="nv">$eth1_in_warn</span><span class="s2">"</span>|bc<span class="sb">`</span>
<span class="nv">eth1_out_diff</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$eth1_out</span><span class="s2"> &lt; </span><span class="nv">$eth1_out_warn</span><span class="s2">"</span>|bc<span class="sb">`</span>
<span class="nv">eth0_in_diff_2</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$eth0_in</span><span class="s2"> > </span><span class="nv">$eth0_in_critical</span><span class="s2">"</span>|bc<span class="sb">`</span>
<span class="nv">eth0_out_diff_2</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$eth0_out</span><span class="s2"> > </span><span class="nv">$eth0_out_critical</span><span class="s2">"</span>|bc<span class="sb">`</span>
<span class="nv">eth1_in_diff_2</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$eth1_in</span><span class="s2"> > </span><span class="nv">$eth1_in_critical</span><span class="s2">"</span>|bc<span class="sb">`</span>
<span class="nv">eth1_out_diff_2</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$eth1_out</span><span class="s2"> > </span><span class="nv">$eth1_out_critical</span><span class="s2">"</span>|bc<span class="sb">`</span>
<span class="k">if</span> <span class="o">[[</span> <span class="nv">$eth0_in_diff_2</span> -eq 1 <span class="o">]]</span> <span class="o">||</span> <span class="o">[[</span> <span class="nv">$eth0_out_diff_2</span> -eq 1 <span class="o">]]</span> <span class="o">||</span> <span class="o">[[</span> <span class="nv">$eth1_in_diff_2</span> -eq 1 <span class="o">]]</span> <span class="o">||</span> <span class="o">[[</span> <span class="nv">$eth1_out_diff_2</span> -eq 1 <span class="o">]]</span>;<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"CRITICAL!The flow are </span><span class="nv">$eth0_in</span><span class="s2">,</span><span class="nv">$eth0_out</span><span class="s2">,</span><span class="nv">$eth1_in</span><span class="s2">,</span><span class="nv">$eth1_out</span><span class="s2"> Kb|eth0_in=</span><span class="k">${</span><span class="nv">eth0_in</span><span class="k">}</span><span class="s2">Kbps;</span><span class="k">${</span><span class="nv">eth0_in_warn</span><span class="k">}</span><span class="s2">;</span><span class="k">${</span><span class="nv">eth0_in_critical</span><span class="k">}</span><span class="s2">;0;0 eth0_out=</span><span class="k">${</span><span class="nv">eth0_out</span><span class="k">}</span><span class="s2">Kbps;</span><span class="k">${</span><span class="nv">eth0_out_warn</span><span class="k">}</span><span class="s2">;</span><span class="k">${</span><span class="nv">eth0_out_critical</span><span class="k">}</span><span class="s2">;0;0 eth1_in=</span><span class="k">${</span><span class="nv">eth1_in</span><span class="k">}</span><span class="s2">Kbps;</span><span class="k">${</span><span class="nv">eth1_in_warn</span><span class="k">}</span><span class="s2">;</span><span class="k">${</span><span class="nv">eth1_in_critical</span><span class="k">}</span><span class="s2">;0;0 eth1_out=</span><span class="k">${</span><span class="nv">eth1_out</span><span class="k">}</span><span class="s2">Kbps;</span><span class="k">${</span><span class="nv">eth1_out_warn</span><span class="k">}</span><span class="s2">;</span><span class="k">${</span><span class="nv">eth1_out_critical</span><span class="k">}</span><span class="s2">;0;0"</span>
<span class="nb">exit </span>2
<span class="k">elif</span> <span class="o">[[</span> <span class="nv">$eth0_in_diff</span> -eq 1 <span class="o">]]</span> <span class="o">[[</span> <span class="nv">$eth0_out_diff</span> -eq 1 <span class="o">]]</span> <span class="o">[[</span> <span class="nv">$eth1_in_diff</span> -eq 1 <span class="o">]]</span> <span class="o">[[</span> <span class="nv">$eth1_out_diff</span> -eq 1 <span class="o">]]</span>;<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"OK!The flow are </span><span class="nv">$eth0_in</span><span class="s2">,</span><span class="nv">$eth0_out</span><span class="s2">,</span><span class="nv">$eth1_in</span><span class="s2">,</span><span class="nv">$eth1_out</span><span class="s2"> Kb|eth0_in=</span><span class="k">${</span><span class="nv">eth0_in</span><span class="k">}</span><span class="s2">Kbps;</span><span class="k">${</span><span class="nv">eth0_in_warn</span><span class="k">}</span><span class="s2">;</span><span class="k">${</span><span class="nv">eth0_in_critical</span><span class="k">}</span><span class="s2">;0;0 eth0_out=</span><span class="k">${</span><span class="nv">eth0_out</span><span class="k">}</span><span class="s2">Kbps;</span><span class="k">${</span><span class="nv">eth0_out_warn</span><span class="k">}</span><span class="s2">;</span><span class="k">${</span><span class="nv">eth0_out_critical</span><span class="k">}</span><span class="s2">;0;0 eth1_in=</span><span class="k">${</span><span class="nv">eth1_in</span><span class="k">}</span><span class="s2">Kbps;</span><span class="k">${</span><span class="nv">eth1_in_warn</span><span class="k">}</span><span class="s2">;</span><span class="k">${</span><span class="nv">eth1_in_critical</span><span class="k">}</span><span class="s2">;0;0 eth1_out=</span><span class="k">${</span><span class="nv">eth1_out</span><span class="k">}</span><span class="s2">Kbps;</span><span class="k">${</span><span class="nv">eth1_out_warn</span><span class="k">}</span><span class="s2">;</span><span class="k">${</span><span class="nv">eth1_out_critical</span><span class="k">}</span><span class="s2">;0;0"</span>
<span class="nb">exit </span>0
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"WARNING!The flow are </span><span class="nv">$eth0_in</span><span class="s2">,</span><span class="nv">$eth0_out</span><span class="s2">,</span><span class="nv">$eth1_in</span><span class="s2">,</span><span class="nv">$eth1_out</span><span class="s2"> Kb|eth0_in=</span><span class="k">${</span><span class="nv">eth0_in</span><span class="k">}</span><span class="s2">Kbps;</span><span class="k">${</span><span class="nv">eth0_in_warn</span><span class="k">}</span><span class="s2">;</span><span class="k">${</span><span class="nv">eth0_in_critical</span><span class="k">}</span><span class="s2">;0;0 eth0_out=</span><span class="k">${</span><span class="nv">eth0_out</span><span class="k">}</span><span class="s2">Kbps;</span><span class="k">${</span><span class="nv">eth0_out_warn</span><span class="k">}</span><span class="s2">;</span><span class="k">${</span><span class="nv">eth0_out_critical</span><span class="k">}</span><span class="s2">;0;0 eth1_in=</span><span class="k">${</span><span class="nv">eth1_in</span><span class="k">}</span><span class="s2">Kbps;</span><span class="k">${</span><span class="nv">eth1_in_warn</span><span class="k">}</span><span class="s2">;</span><span class="k">${</span><span class="nv">eth1_in_critical</span><span class="k">}</span><span class="s2">;0;0 eth1_out=</span><span class="k">${</span><span class="nv">eth1_out</span><span class="k">}</span><span class="s2">Kbps;</span><span class="k">${</span><span class="nv">eth1_out_warn</span><span class="k">}</span><span class="s2">;</span><span class="k">${</span><span class="nv">eth1_out_critical</span><span class="k">}</span><span class="s2">;0;0"</span>
<span class="nb">exit </span>1
<span class="k">fi</span>
</code></pre>
</div>
concat
2010-08-04T00:00:00+08:00
CDN
http://chenlinux.com/2010/08/04/concat
<p>今天逛到淘宝核心系统组的公开博客,然后知道了淘宝开源平台<a href="http://code.taobao.org/">http://code.taobao.org/</a>,刚出来的东东,上面开源了淘宝目前正在使用的key-value分布式存储tair和nginx的concet模块、Cookie模块,以及这个平台本身的代码……在回复中,淘宝的人说有关tair的update都会同步在这个平台上进行,而不是另起一个闭源分支,或者想新浪的sina-sdd那样放弃。据说九月份,淘宝还会放出它的分布式文件系统TFS来,毕竟时间不长,会不会“人亡政息”,有待考验……</p>
<p>不过我对这个concet模块有点兴趣,从其英文介绍(淘宝真是,对国人也用英文)来看,是仿照apache的mod_concet用来合并js/css文件输出的。于是去找mod_concet资料来看。其官方发布在google上。说明很简单,安装方法、配置方法、效果……<br />
<code class="highlighter-rouge">css
<link href=/a/a.css type=text/css>
<link href=/a/b.css type=text/css>
<link href=/a/c.css type=text/css>
</code><br />
统一写成<link href=/a/a.css,/a/b.css,/a/c.css type=text/css>就可以了。<br />
效果据说快20%-30%。<br />
写mod_concet模块的AOL工程师自己的截图如下:<br />
<img src="http://pic04.babytreeimg.com/foto/thumbs/59/14/67/3472f5340b8860e7e8f4_m.png" alt="" /><br />
其last-modified定义,则是选取了合并前的文件中MTIME最晚的那个。<br />
或许有些道理,因为浏览器对同一域名能开启的并发数有限,而太小的文件下载速度加不起来……<br />
不过js或者css不一定总是能写在一块的,留作一个记录吧~~</p>
squid2.6的bug(if-none-match)
2010-08-03T00:00:00+08:00
CDN
squid
http://chenlinux.com/2010/08/03/squid2-6-bug-about-if_none_match
<p>话接上篇。<br />
在发现经过squid的刷新请求头不带If-None-Match后,向squid-user发邮件询问。Amos回复如下:</p>
<p>On Mon, 02 Aug 2010 07:56:09 +0800, Gemmy <a href="mailto:chenryn@163.com"><span style="text-decoration:underline;"><span style="color:#0000ff;"><a href="mailto:chenryn@163.com">chenryn@163.com</a></span></span></a> wrote:</p>
<blockquote style="color:#000000;">
Hi~
> I have a nginx webserver adding the static-etags module and a
> squid2.6.23 before the nginx. When I access to the nginx directly, I can
> see the ETag in response header and If-None-Match in the request header
> after refresh. But when I access to the squid, ETag header still
> exist,If-None-Match header disappeared!
> The squid.conf have most basic configuration items, nothing about cache.
> Why does this happen?
</blockquote>
<p>Squid-2 has problems with If-None-Match.<br />
<a href="http://bugs.squid-cache.org/show_bug.cgi?id=2112"><span style="text-decoration:underline;"><span style="color:#0000ff;">http://bugs.squid-cache.org/show_bug.cgi?id=2112</span></span></a></p>
<p>Please upgrade to Squid-2.7. It will behave a lot better regarding ETags<br />
in general and <a href="http://bugs.squid-cache.org/show_bug.cgi?id=2112"><span style="text-decoration:underline;"><span style="color:#0000ff;">http://bugs.squid-cache.org/show_bug.cgi?id=2112</span></span></a> has some<br />
patches on 2.7 that you may want to try.</p>
<p>Amos</p>
<p>赶紧看看bug2112报告《Squid does not send If-None-Match tag for cache revalidation》,原来是http.c里,虽然<br />
httpBuildRequestHeader定义了if-none-match,但却忘了定义接下来向源站发送的request-etags!<br />
重新下载squid2.7.9,编译完成后开始测试。访问动作和相应的日志记录如下:<br />
第一次访问:<br />
1280803634.566 65 59.108.42.242 TCP_MISS/200 19127 GET <a href="http://club.china.com/data/thread/1011/2716/16/81/5_1.html">http://club.china.com/data/thread/1011/2716/16/81/5_1.html</a> - ROUNDROBIN_PARENT/127.0.0.1 text/html “<a href="http://club.china.com/">http://club.china.com/</a>” “Mozilla/5.0 (Windows; Windows NT 5.1; rv:2.0b2) Gecko/20100720 Firefox/4.0b2”<br />
F5刷新有新内容:<br />
1280803714.220 488 59.108.42.242 TCP_REFRESH_MISS/200 19127 GET <a href="http://club.china.com/data/thread/1011/2716/16/81/5_1.html">http://club.china.com/data/thread/1011/2716/16/81/5_1.html</a> - ROUNDROBIN_PARENT/127.0.0.1 text/html “<a href="http://club.china.com/">http://club.china.com/</a>” “Mozilla/5.0 (Windows; Windows NT 5.1; rv:2.0b2) Gecko/20100720 Firefox/4.0b2”<br />
F5刷新没新内容:<br />
1280805207.149 3 59.108.42.242 TCP_REFRESH_HIT/304 309 GET <a href="http://club.china.com/data/threads/1011/1.html">http://club.china.com/data/threads/1011/1.html</a> - ROUNDROBIN_PARENT/127.0.0.1 text/html “-“ “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)”<br />
回复后返回新内容:<br />
1280803803.696 20 59.108.42.242 TCP_REFRESH_HIT/200 19126 GET <a href="http://club.china.com/data/thread/1011/2716/16/81/5_1.html">http://club.china.com/data/thread/1011/2716/16/81/5_1.html</a> - ROUNDROBIN_PARENT/127.0.0.1 text/html “<a href="http://r.club.china.com/jsp/pub/replyAutoJump.jsp?url=http://club.china.com/data/thread/1011/2716/16/81/5_1.html&forumid=1011&message=&picmessage">http://r.club.china.com/jsp/pub/replyAutoJump.jsp?url=http%3A%2F%2Fclub.china.com%2Fdata%2Fthread%2F1011%2F2716%2F16%2F81%2F5_1.html&forumid=1011&message=&picmessage</a>=” “Mozilla/5.0 (Windows; Windows NT 5.1; rv:2.0b2) Gecko/20100720 Firefox/4.0b2”<br />
Ctrl+F5刷新:<br />
1280803821.731 12 59.108.42.242 TCP_CLIENT_REFRESH_MISS/200 19127 GET <a href="http://club.china.com/data/thread/1011/2716/16/81/5_1.html">http://club.china.com/data/thread/1011/2716/16/81/5_1.html</a> - ROUNDROBIN_PARENT/127.0.0.1 text/html “<a href="http://r.club.china.com/jsp/pub/replyAutoJump.jsp?url=http://club.china.com/data/thread/1011/2716/16/81/5_1.html&forumid=1011&message=&picmessage">http://r.club.china.com/jsp/pub/replyAutoJump.jsp?url=http%3A%2F%2Fclub.china.com%2Fdata%2Fthread%2F1011%2F2716%2F16%2F81%2F5_1.html&forumid=1011&message=&picmessage</a>=” “Mozilla/5.0 (Windows; Windows NT 5.1; rv:2.0b2) Gecko/20100720 Firefox/4.0b2”</p>
郁闷的last-modified和etag试验
2010-08-01T00:00:00+08:00
CDN
nginx
SSI
http://chenlinux.com/2010/08/01/test-last_modified-etag-of-nginx
<p>手上有一个伪静态的论坛bbs.example.com,每当有用户回复帖子的时候,提交到r.bbs.example.com,然后由r.bbs.example.com的提示页面replyautojump.jsp延迟3000后调用window.location.href函数返回原来的帖子——期间即完成对该帖的静态化。</p>
<p>为了实效性,之前的配置,将这类html都设定为no-cache。现在考虑到流量和IO的问题,计划希望能够将这类html尽可能的缓存在browser和proxy上,但每次都能采用304的方式,确认帖子的最新情况。对于不太热门的板块,不太热点的帖子,应该能缓存一段时间,减轻webserver的压力;就算是热门,哪怕几十秒钟的缓存,对于全网流量,也是积腋成裘。。。</p>
<p>因为论坛页面上,用SSI方式include了一些广告、点击排名等挂件。所以nginx上设定了ssi on,导致response-header中,没有了last-modified——按照常规,一般是在长期不变的html里include经常改变的shtml,而每次在解析SSI时去读取所有shtml的MTIME然后计算最后一个MTIME来设定成总页面的last-modified,是个比较繁琐且耗资源的做法(网上有lighttpd的patch,就是这么做的),所以包括nginx在内的多数webserver采取了比较简便的方法,即取消掉last-modified输出。参见nginx/src/http/module/ngx_http_ssi_filter_module.c第361行:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">static</span> <span class="n">ngx_int_t</span>
<span class="nf">ngx_http_ssi_header_filter</span><span class="p">(</span><span class="n">ngx_http_request_t</span> <span class="o">*</span><span class="n">r</span><span class="p">)</span>
<span class="p">{</span>
<span class="err">……</span>
<span class="k">if</span> <span class="p">(</span><span class="n">r</span> <span class="o">==</span> <span class="n">r</span><span class="o">-></span><span class="n">main</span><span class="p">)</span> <span class="p">{</span>
<span class="n">ngx_http_clear_content_length</span><span class="p">(</span><span class="n">r</span><span class="p">);</span>
<span class="n">ngx_http_clear_last_modified</span><span class="p">(</span><span class="n">r</span><span class="p">);</span>
<span class="n">ngx_http_clear_accept_ranges</span><span class="p">(</span><span class="n">r</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">ngx_http_next_header_filter</span><span class="p">(</span><span class="n">r</span><span class="p">);</span>
<span class="p">}</span>
</code></pre>
</div>
<p>而我这系统的情况,可能广告1天一变,排行15分钟一变,都远远慢于页面本身的更新速度,完全可以将html的MTIME认定为总页面的last-modified。<br />
注释掉相应源码,重新编译nginx后进行试验,在使用F5刷新的时候,果然发送了IMS,这一步成功了;可是回复帖子后,因为是跳转方式,浏览器直接采用本地cache,而不会发送IMS,所以还是看到旧页面。。。</p>
<hr />
<p>因为测试平台上正好有另一个端口运行着httpd,不小心发现在跳转时如果有etag的话,还是会发送INM确认。于是今天开始试验往nginx上加etag。<br />
从git上下载etag模块源码,采用add-module方式重编译nginx,启动后先试验直接从nginx访问。确认其在回复跳转时发送了INM,然后加上前段squid,怪事出现了——虽然response-header中确实有etag,刷新网页时request-header中却一直没有出现if-none-match!!</p>
<p>返回查看squid和nginx的日志。squid中一直记录的是TCP_REFRESH_MISS/200,nginx上更离谱!304时传输的文件大小居然比200时还大!见下:<br />
127.0.0.1 - - [01/Aug/2010:15:18:18 +0800] “GET /data/thread/1011/2716/14/71/9_1.html HTTP/1.0” 200 14823 “-“ “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)” “124.200.241.143”<br />
124.200.241.143 - - [01/Aug/2010:15:18:57 +0800] “GET /data/thread/1011/2716/14/71/9_1.html HTTP/1.1” 304 23459 “-“ “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)” “-“</p>
<hr />
<p>最后作为尝试,再打开SSI的last-modified同时加载etag编译了一次,试验结果,发现和没etag时一模一样,只见IMS不见INM。可是squid明明在2.6.1开始就支持etag了,怪哉~~</p>
<p>下次有时间,再试试lighttpd的etag吧~</p>
网络监控与/proc/net/tcp文件
2010-07-28T00:00:00+08:00
monitor
procfs
http://chenlinux.com/2010/07/28/monitor-netflow-by-proc_net_tcp
<p>nagios自带的check_antp太过简约,除了状态统计输出外,什么参数都不提供。在面对不同应用服务器时,报警就成了很大问题。于是决定自己写一个check脚本。作脚本运行,与命令操作时一个不同,就是要考虑一下效率问题。在高并发的机器上定期运行netstat -ant命令去统计,显然不太合适,可以直接从proc系统中取数据,这就快多了。</p>
<p>先介绍/proc/net/tcp文件,这里记录的是ipv4下所有tcp连接的情况,包括下列数值:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
0: 00000000:3241 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 22714864 1 ffff88004f918740 750 0 0 2 -1
</code></pre>
</div>
<p>最主要的,就是local_address本地地址:端口、rem_address远程地址:端口、st连接状态。</p>
<p>注1:文件中都是用的16进制,所以HTTP的80端口记录为0050。 <br />
注2:状态码对应如下</p>
<div class="highlighter-rouge"><pre class="highlight"><code>00 "ERROR_STATUS",
01 "TCP_ESTABLISHED",
02 "TCP_SYN_SENT",
03 "TCP_SYN_RECV",
04 "TCP_FIN_WAIT1",
05 "TCP_FIN_WAIT2",
06 "TCP_TIME_WAIT",
07 "TCP_CLOSE",
08 "TCP_CLOSE_WAIT",
09 "TCP_LAST_ACK",
0A "TCP_LISTEN",
0B "TCP_CLOSING",
</code></pre>
</div>
<p>然后介绍nrpe的check脚本。脚本不管怎么写都行,对于nagios服务器端来说,它除了接受脚本的输出结果外,只认脚本运行的退出值(测试时可以运行后用echo $?看),包括OK的exit 0、WARNING的exit 1、CRITICAL的exit 2、未知的exit 3。</p>
<p>最后一个简单的检查http端口连接数的脚本如下:<br />
```bash<br />
#!/bin/bash<br />
#Written by Gemmy.Rao<br />
#Email to: <a href="mailto:chenlin.rao@bj.china.com">chenlin.rao@bj.china.com</a><br />
#Version 0.2<br />
#CHANGES<br />
#Add -p option for checking other service’s port</p>
<p>#Init<br />
PORT=80<br />
WARNING=5000<br />
CRITICAL=20000</p>
<p>#get options<br />
while getopts “w:c:p:hs” OPT;do<br />
case $OPT in<br />
w)<br />
WARNING=${OPTARG}<br />
;;<br />
c)<br />
CRITICAL=${OPTARG}<br />
;;<br />
p)<br />
PORT=${OPTARG}<br />
#转换各端口的十进制成十六进制<br />
PORT_16=<code class="highlighter-rouge">echo ${PORT}|awk -F, '{for(i=1;i<=NF;i++)printf "|%.4X",$i}'|sed 's/|//'</code><br />
;;<br />
h)<br />
echo “Usage: $0 -w 500 -c 2000 -p 80,8081 -s”<br />
exit 0<br />
;;<br />
s)<br />
SILENT=1<br />
;;<br />
*)<br />
echo “Usage: $0 -w 500 -c 2000 -p 80,8081”<br />
exit 0<br />
;;<br />
esac<br />
done</p>
<p>#经过time测试,取值速度netstat > awk ‘//{a++}END{print a}’ > cat|grep|wc > cat|awk|wc,在2w连接下,netstat要20s,最快的方式不到5s(一般nagios到10s就该直接报timeout了)<br />
PORT_CONN=<code class="highlighter-rouge">cat /proc/net/tcp*|awk '$2~/:('$PORT_16')$/'|wc -l</code></p>
<p>if [[ “$SILENT” == 1 ]];then<br />
[[ -d /usr/local/nagios ]] || mkdir -p /usr/local/nagios<br />
echo “Silent log write OK | Port ${PORT}=${PORT_CONN};${WARNING};${CRITICAL};0;0”<br />
echo -en “<code class="highlighter-rouge">date</code>t$PORT_CONNn” » /usr/local/nagios/conn.log<br />
exit 0<br />
elif [[ “$PORT_CONN” -lt “$WARNING” ]];then<br />
echo “Port $PORT connection OK for $PORT_CONN. | Port ${PORT}=${PORT_CONN};${WARNING};${CRITICAL};0;0”<br />
exit 0<br />
elif [[ “$PORT_CONN” -gt “$CRITICAL” ]];then<br />
echo “Port $PORT connection critical for $PORT_CONN!! | Port ${PORT}=${PORT_CONN};${WARNING};${CRITICAL};0;0”<br />
exit 2<br />
else<br />
echo “Port $PORT connection warning for $PORT_CONN! | Port ${PORT}=${PORT_CONN};${WARNING};${CRITICAL};0;0”<br />
exit 1<br />
fi<br />
```</p>
<p>之后有必要的话,可以再取$4去统计st。</p>
日志流量计算
2010-07-28T00:00:00+08:00
bash
awk
http://chenlinux.com/2010/07/28/compute-flows-by-access_log
<p>一般来说流量带宽是通过snmp协议取网卡流量画图。不过有的时候,为了优化分析或者排错,也会直接去计算服务的访问流量。方法很简单,根据日志中记录的请求时间(squid记录的是请求响应完成时间,如果要精确,可以再减去响应时间,不过一般squid的文件不至于5分钟内还传不完的……),按每5分钟一汇总其字节数,然后均摊到300秒上。</p>
<p>计算全日志中最高带宽的命令行如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>cat <span class="nv">$ACCESS_LOG</span>|awk -F<span class="s1">'[: ]'</span> <span class="s1">'{a[$5":"$6]+=$14}END{for(i in a){print i,a[i]}}'</span>|sort|awk <span class="s1">'{a+=$2;if(NR%5==0){if(a>b){b=a;c=$1};a=0}}END{print c,b*8/300/1024/1024}'</span>
</code></pre>
</div>
<p>(日志为标准apache日志格式)<br />
而把最后的awk改成’{a+=$2;if(NR%5==0){print $1,a*8/300/1024/1024;a=0}}’,就可以输出每5分钟时的流量值,然后用GD库画图~~(有时间看看perl的GD:Graph模块,应该不难)</p>
域名切换导致的 SEO 问题
2010-07-17T00:00:00+08:00
web
http://chenlinux.com/2010/07/17/location-seo
<p>网站某频道准备启用新域名,切换过程中,为了保证网民访问效果,对老域名下的所有请求设置了重定向到新域名下相同url。测试访问正常后上线使用。</p>
<p>过几天,发现该频道在各搜索引擎的收录数和关键词排名中都消失了!</p>
<h2 id="se0-">SE0 收录的问题</h2>
<p>原来对于搜索引擎来讲,301永久重定向与302临时重定向属于不同情况。它们认可301永久重定向规则,并依此规则转移原请求的数据……</p>
<p>照此修改nginx配置</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">server</span> <span class="p">{</span>
<span class="kn">server_name</span> <span class="s">old.domain.com</span><span class="p">;</span>
<span class="c1">#rewrite ^/(.*)$ http://new.domain.com/$1 last;
</span> <span class="kn">rewrite</span> <span class="s">^/(.*)</span>$ <span class="s">http://new.domain.com/</span><span class="nv">$1</span> <span class="s">permanent</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">server</span> <span class="p">{</span>
<span class="c1">#server_name old.domain.com toold.domain.com;
</span> <span class="kn">server_name</span> <span class="s">new.domain.com</span> <span class="s">toold.domain.com</span><span class="p">;</span>
<span class="kn">root</span> <span class="n">/www/old.domain.com</span><span class="p">;</span>
<span class="kn">index</span> <span class="s">index.html</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
<p>配置生效后,收录逐渐恢复。</p>
<p>目前查询的结果,新老域名的google收录比为28600:46100,百度收录比为1200:97400。</p>
<p>据网友经验,百度对301重定向大概也需要3个月左右的时间才能完全反应过来……汗死</p>
<h2 id="google-pr">google PR的问题</h2>
<p>收录解决后,PR的问题又报出来了。目前查询结果,old.domain.com的PR为6,new.domain.com的PR还是0,而且toolod.domain.com的PR也是6,并提示可能是劫持了old.domain.com的PR!</p>
<p>只好再去看PR的资料……PR是google发明的对页面重要性等级的分析算法。从0到10非等比升高。主要是通过相互之间的链接来衡量得出的,外部链接的PR越高,网站得到的PR评分也越高(粗略的说)。对于google来说,google、yahoo等搜索引擎的收录,显然是PR衡量中极为重要的(google自定为10,yahoo、baidu等搜索是9,sina等门户是8…)显然,对于网站域名迁移切换来说,搜索引擎的收录数迁移是基础。</p>
<p>不过收录转移完了,PR也变不了——因为PR计算太复杂了,哪怕以google的实力也不可能做到实时更新,而是几乎2.5-3个月才更新一次!</p>
<p>同时,据google的Webspam团队老大Matt Cutts说:即使在域名迁移中使用301重定向,在PR统计时,也会有一定的损失!</p>
<p>总之,想看到new.domain.com的PR恢复成6,耐心等待吧……</p>
<p>然后研究toold的劫持PR报告是怎么来的……</p>
<p>一般大家的说法,劫持PR都是采用重定向或者别名的方式(上面提到了,这个方式也不会获得完全一样的PR)。显然我这里的情况不是。</p>
<p>也有人说,劫持PR直接A到IP更方便。也有人怀疑自己租机建站得到的是其他站的PR,难道相同IP的域名PR都会一样?查询一下和new.domian.com在同一台nginx上的另一个域名,PR是5。猜测不对。</p>
<p>或许,因为toold读取的网页文件和old是一致的,所以获得的外链也一致。但外面的反向链接,肯定指的都是old而不是toold,导致toold的反链数过少,所以被怀疑为PR劫持了。</p>
<p>不过,据编辑说,toold这个域名从来就没有上线发布使用过,搜索引擎从哪里抓到的页面呢?</p>
<p>nginx 的同一 server{} 段内的多个 <code class="highlighter-rouge">server_name</code> 配置,默认只会把第一个域名设定为 <code class="highlighter-rouge">$server_name</code>。</p>
<p>dns 上也没有配反向解析。</p>
<p>实在想不到还有哪里能泄露出这个toold了……</p>
初识NetApp存储
2010-07-17T00:00:00+08:00
linux
NetApp
http://chenlinux.com/2010/07/17/intro-netapp
<p>对专门的存储设备实在所知甚少,今天有时间找了找netapp的资料看看,有点基本了解。资料见豆丁:<a href="http://www.docin.com/p-56454923.html">http://www.docin.com/p-56454923.html</a></p>
<p>从系统运维角度来说,首先关注的是RAID部分,netapp自定义有RAID-DP。这是一个私有的RAID6解决方案,在RAID4的基础上发展而来的。 <br />
RAID4即专有一个同位数据校验盘,原始数据以块大小分割存储; <br />
RAID5则将同位校验数据和原始数据重新组合,以位大小分割存储; <br />
RAID6标准是在存放同位校验数据同时,还增加了针对块的校验数据; <br />
RAID-DP,同时拥有两种校验,并把这两种校验都单独存盘。 <br />
即,RAID-DP,每组至少需要3个盘;而据资料显示,netapp最多支持每组28个盘,默认则是16个。</p>
<p>多个RAID组组成一个Plex,然后plex组成aggregate。从资料上看,aggr中的plex一般不多。这些硬件上的情况,可以用aggr status -r来查看。 <br />
在aggr上,再进行逻辑卷volume的划分,一个flexvol最小20MB,最大16T,可以4K大小的加减;最多可以创建500个flexvol。</p>
<p>然后是使用方式,DAS/NAS/SAN都行,现在在用的是NFS挂载(NAS),以后或许试试iscsi(SAN)。这部分内容看之前相关博文就行。</p>
<p>IO方式,分file和blockI/O两种。</p>
<p>监控方式,netapp自带一些性能监控命令,sysstat、nfsstat、netstat等等。</p>
<hr />
<p>在存储设备选择比较时,最经常被提及的一个参数是IOPS,即每秒IO操作数。 <br />
在netapp上运行sysstat命令,看到最前列有个ops/s,峰值高达15k。这个数值被设备方的销售认为是绝对不可能的……</p>
<p>于是抠字眼吧,iops和ops有什么不同呢?有文档说ops叫每秒并发操作数,并不限定是IO。不过作为专业存储设备,基本除了IO也不会有什么别的操作了吧?</p>
<p>正好,因为图省事,我比较喜欢看设备的web界面性能监控页自动画出的柱状图。第二个图就是ops/s的图,图左的文字说明是“Network File Operate per second“,即每秒网络文件操作数。而询问销售,IOPS却是指的block的操作数——区别就在这里!</p>
<p>因为采用NFS挂载存储的方式无法直接观察block的操作监控,最后采用一个估算的办法,统计每秒IO的文件大小,然后除以block的大小,得出每秒IO操作数。 <br />
还是用systat命令,取max的r/s和w/s(KB)值,相加得32k。 <br />
然后用aggr status -b命令,取得block值为4KB,估算得IOPS为8k。 <br />
还是用aggr status -r命令,取得disk的总数为42,估算每块盘的IOPS为180。 <br />
和网上的评论比较,一般测试在170多的时候性能最好,再高就下降。看来这个设备运行在比较悬的状态了——目前CPU监测使用率在95%左右。</p>
<p>存储上的磁盘都是15000转的,在IO发生时,磁盘要转动,机头要移动,都需要时间,开始计算: <br />
15000/60=250PRS <br />
1/250=0.004spr=4mspr <br />
4/2=2ms(磁盘转动耗时RD,最多半圈) <br />
2+4=6ms(机头寻道时间,百度一下,希捷3.4,日立3.3,都是最快的那种,就按4ms算吧) <br />
1000/6=167IOPS(按日立那个最快的算是188)</p>
<p>写到最后,在CU上看到一个好帖子,链接如右:<a href="http://bbs.chinaunix.net/thread-1607334-1-1.html">http://bbs.chinaunix.net/thread-1607334-1-1.html</a></p>
login-shell的更改
2010-07-06T00:00:00+08:00
linux
http://chenlinux.com/2010/07/06/modify-login-shell
<p>之前用ports把BSD系统的login-shell改成bash后,今天又打算改回csh去。不料重新chsh -s /bin/csh后,却弹出如下错误提示:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>chsh: entry inconsistent
chsh: pw_copy: Invalid argument
</code></pre>
</div>
<p>很诡异的提示~然后直接运行chsh修改SHELL;修改/etc/passwd和/etc/master.passwd中的/usr/local/bin/bash为/bin/csh;退出重登录,还是bash没变……</p>
<p>经过谷大婶的帮助,发现原来在login的时候,并不是读取/etc/master.passwd来决定login-shell,而是会去读取/etc/pwd.db和/etc/spwd.db。只需要运行如下命令更新这两个db文件就可以了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>pwd_mkdb /etc/master.passwd
</code></pre>
</div>
<p>试验一下,果然ok了~~</p>
Java报错一例
2010-07-03T00:00:00+08:00
java
http://chenlinux.com/2010/07/03/a-memory-error-of-java
<p>新迁移一例apache+resin系统,运行不久后时不时就出现500错误。从日志中看到如下报错:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>OutOfMemoryError: PermGen space
</code></pre>
</div>
<p>内存溢出,可是free、top来看,mem和swap都还有很多空闲~<br />
于是去谷歌一下,看到如下说法:</p>
<p>PermGen space全称Permanent Generation space(永久保存内存区),这部分内存用来保存Class和Meta的信息——Class在load的时候进入PermGen,之后Java运行时,GC(垃圾回收机制)就不再去管这部分内容了。当resin对jsp进行percompile时,可能就导致内存溢出了……默认空间为4M!<br />
Heap space(JVM运行调用的内存区),JVM启动时,默认的初始空间Xms是物理内存的1/64,最大空间Xmx是物理内存的1/4。建议指定Xms和Xmx相同(小于物理内存的80%),然后Xmn为Xmx的1/4。</p>
<p>解决办法:</p>
<p>修改java启动参数,追加“-Xms256m -Xmx256m -XX:MaxNewSize=256m -XX:MaxPermSize=256m”,定死space大小。<br />
重启至今故障未出现。</p>
动态页面正文部分中文乱码排障一例
2010-06-26T00:00:00+08:00
web
apache
nginx
resin
http://chenlinux.com/2010/06/26/example-of-chinese-garbled-in-dynamic-page
<p>公司网站一部分动态页面,早先使用apache+resin的架构运行,考虑到高并发访问下的响应性能问题,在前不久逐步开始用nginx替换掉了apache。 <br />
不过随后发现了一个问题,随意进入某一有分页的网页,第一页是正常的(因为静态化过了);点“下一页”,出来的页面两边正常,中间部分的标题、关键字等也正常,唯独每个标题下的正文无法正常显示。 <br />
因为有做过系统调整,所以第一反应就是新上的nginx配置有问题。按照经验,可能是nginx.conf中指定的chaset与borwser不一致?但选定utf8后现象依旧,何苦同一页面内的其他字符又是正确显示的~~~ <br />
然后通过内网IP+端口的方式,直接向resin请求抓取到的乱码页面url。结果,nginx+resin的机器显示乱码,apache+resin的机器显示中文——由此确认问题不是nginx,而是resin的! <br />
diff两台机器的resin.conf,除了开启的端口外,没有任何不同的地方。 <br />
检查两台机器的环境变量,发现nginx这台的LANG是zh_CDN:gbk(静态化程序有需求),而apache这台是utf8。试着也修改成utf8然后重启resin,访问结果依然不对。 <br />
这下基本没招了……完全一样的环境和配置,取的同一台nfs的数据,为啥就能显示不同呢?难道是编译参数的问题? <br />
去sharepoint上下载公司文档,查看原先的resin都使用了那些configure选项。结果发现为了配合apache,使用了–with-apache等。莫非就是因为这个原因导致resin脱离apache运行出现问题了? <br />
下载和现行resin版本一致的源码报,不再with-apache编译完成,cp一份conf过来,改用另一个端口启动,然后通过这个端口访问那个url,结果显示正常了! <br />
替换下原先的resin,把nginx的upstream指向新resin,故障解决。 <br />
看来以后再替换apache+resin成nginx+resin的时候,resin也要重新编译一个了……</p>
$RANDOM变量妙用
2010-06-23T00:00:00+08:00
bash
http://chenlinux.com/2010/06/23/intro-random-variable
<p>$RANDOM是linux自带的一个随机数变量,其随机范围从0-32767(man bash说的)。每次unset再恢复后,$RANDOM都会变化。</p>
<p>那么,在想获得某个范围内的随机数的时候,只需要很简单的利用一下这个变量就可以了。比如想随机生成一个C段的ip。如下:</p>
<p>echo 192.168.0.$[$RANDOM*255/32767]<br />
192.168.0.71</p>
折腾 awk 内调用 shell 变量
2010-06-07T00:00:00+08:00
bash
awk
http://chenlinux.com/2010/06/07/testing-awk-shell-variables
<p>在对squid进行目录刷新的时候,一般使用的脚本都是采用for i in <code class="highlighter-rouge">squidclient mgr:objects|grep $1|awk '{print $2}'</code>;do squidclient -m purge “$i”;done的方式。</p>
<p>mgr:objects本来就是一个比较费资源的请求,假如一个200G的cache,这个$i变量该卡多久才能有反应?抑或直接挂掉……</p>
<table>
<tbody>
<tr>
<td>于是把这个稍微改进一下,变成squidclient mgr:objects</td>
<td>awk ‘/”‘$1’”/{system(“squidclient -m purge “$2)}’,因为awk对每行进行匹配后,就可以同时作出反应,所以比存一个大变量要好一些。</td>
</tr>
</tbody>
</table>
<p>不过在使用中发现还有一些别的问题。比如碰到http://www.test.com/abc(123).html这样的url的时候,就会出错:</p>
<p>sh: -c: line 0: syntax error near unexpected token `(‘</p>
<p>url里的括号和awk函数的括号冲突了。所以对$2不能简单引用就完,还得处理。<br />
CU上有人给出如下写法:</p>
<p>awk ‘{system(“squidclient -m purge ‘’’“$2”’’’”)}’</p>
<p>一试果然可以,试着分解一下这堆引号:</p>
<p>‘{system(“squidclient -m purge ‘第一部分,单引号表示里面的内容都传递给awk处理;</p>
<p>‘第二部分,shell环境下转义单引号为普通字符;</p>
<p>’“$2”‘第三部分,传递给awk,其中第一个”接第一部分的”,完成system函数的命令部分,其中包括了第二部分的普通字符’;</p>
<p>‘第四部分,shell环境下转义单引号为普通字符;</p>
<p>’”)}’第五部分,传递给awk,其中”接第三部分的第二个”,其中包含了第四部分的普通字符’;</p>
<p>合在一起,就给替换好的$2加上了一对’‘,然后通过system函数传递给shell执行。OK~~</p>
文件锁和 CPU 绑定
2010-05-30T00:00:00+08:00
linux
taskset
flock
http://chenlinux.com/2010/05/30/intro-two-lock-tools
<p>从网上看到的两个锁定,都是util-linux包里的。</p>
<p>一个是flock。我从字面上猜测是文件描述符锁~~help显示如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>flock (util-linux 2.13-pre7)
Usage: flock [-sxun][-w #] fd#
flock [-sxon][-w #] file [-c] command...
-s --shared Get a shared lock
-x --exclusive Get an exclusive lock
-u --unlock Remove a lock
-n --nonblock Fail rather than wait
-w --timeout Wait for a limited amount of time
-o --close Close file descriptor before running command
-c --command Run a single command string through the shell
-h --help Display this text
-V --version Display version
</code></pre>
</div>
<p>比如在rsync定时同步某文件夹的时候,可能担心上一次任务还没执行完,下一次就开始了。于是可以采用如下方式:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1 * * * * flock -xn /var/run/rsync.lock -c 'rsync -avlR /data/files 172.16.xxx.xxx:/data'
</code></pre>
</div>
<p>对照usage,x创建一个独享锁,n是如果已存在就退出(这点扶凯说是就等待,但我觉得从help来看是退出,然后等下一分钟重新探测),然后一个lock文件,c是shell命令,具体内容就是rsync。<br />
另一个是taskset,同样字面来看,任务设定锁。help如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>taskset (util-linux 2.13-pre7)
usage: taskset [options] [mask | cpu-list] [pid | cmd [args...]]
set or get the affinity of a process
-p, --pid operate on existing given pid
-c, --cpu-list display and specify cpus in list format
-h, --help display this help
-v, --version output version information</p>
The default behavior is to run a new command:
taskset 03 sshd -b 1024
You can retrieve the mask of an existing task:
taskset -p 700
Or set it:
taskset -p 03 700
List format uses a comma-separated list instead of a mask:
taskset -pc 0,3,7-11 700
Ranges in list format can take a stride argument:
e.g. 0-31:2 is equivalent to mask 0x55555555
</code></pre>
</div>
<p>用这个命令,可以把不同的进程,锁定在不同的CPU上完成。这个做法,之前在nginx优化上曾经碰到过,不过那是nginx自带的功能。<br />在CU上看到有人提起squid与CPU,squid是只支持单CPU的,不过可以通过在不同端口开启多squid进程的办法来完成对多CPU的利用,一般情况下,各squid进程会自动分配在不同CPU上跑,不过这个是系统的资源分配,难保出问题,就可以用这个命令完成锁定了。</p>
xen安装PV
2010-05-28T00:00:00+08:00
cloud
xen
http://chenlinux.com/2010/05/28/xen-install-pv
<p>闲来无事,打算自己安装一个xen虚拟机,看了看文档,知道必须采用网络安装方式(NFS/FTP/HTTP),于是随手去搜狐镜像站下了一个iso下来挂载用。<br />
不过virt-install一直报错。<br />
首先是:<br />
mount.nfs: Input/output error<br />
umount: /var/lib/xen/xennfs.jfkgaj: not mounted<br />
ERROR: Unable to mount NFS location!<br />
诡异了,我手动都能mount上远端的nfs了~~百度没结果,谷大婶出动,原来这边也要启动portmap才行。<br />
下一步,继续出错:<br />
ERROR: Invalid NFS location given: [Errno 2] No such file or directory: ‘/var/lib/xen/xennfs.JjVbzO/images/xen/vmlinuz’<br />
没有文件?返回nfs上去看,嗯,目录下只有一个LiveCD,一个isolinux。咋回事呢?<br />
又返回搜狐去翻目录,在os/下看到了images/xen/vmlinuz,难道要把整个os/目录下载了?可我记得这个目录就应该是iso挂载后的东西呀~<br />
返回isos/去看,终于发现一个极弱智的问题:目录下有LiveCD和bin-DVD两个镜像,我直接点了最顶上的一个,也就是LiveCD那个……<br />
赶紧重新下载……<br />
之后一路顺利。<br />
A机(10.10.10.10)上:<br />
<code class="highlighter-rouge">bash
wget http://mirrors.sohu.com/centos/5.4/isos/x86_64/CentOS-5.4-x86_64-bin-DVD.iso -c
mount -o loop -t iso9660 /cache/CentOS-5.4-x86_64-bin-DVD.iso /mnt
echo '/mnt 10.10.10.0/24(ro,async)'>>/etc/exports
/etc/init.d/portmap start
/etc/init.d/nfs start
</code><br />
B机(10.10.10.11)上:<br />
<code class="highlighter-rouge">bash
mkdir /img
dd if=/dev/zero of=/img/test.img bs=1024k count=8k
virt-install --paravirt --file=/img/test.img --name=test --ram=1024 --vcpus=1 --bridge=xenbr0 --bridge=xenbr1 --nographics --location=nfs:10.10.10.10:/mnt
</code><br />
(半虚拟化、虚拟机安装位置、虚拟机名、内存、CPU、桥接网卡*2、文本模式、安装源)<br />
然后就是很普通的linux安装过程了,填ip,分区云云……选择最小化安装,reboot。<br />
又见报错:<br />
Restarting system.<br />
libvir: Xen Daemon error : GET operation failed: <br />
Guest installation complete… restarting guest.<br />
libvir: Xen Daemon error : GET operation failed: <br />
libvir: Xen Daemon error : internal error domain information incomplete, missing kernel<br />
Entity: line 30: parser error : Opening and ending tag mismatch: os line 5 and domain<br />
</domain><br />
^<br />
Entity: line 31: parser error : Premature end of data in tag domain line 1<br />
哪里有问题呢?<br />
随手xm list,发现居然有一个名叫test的的guestOS,赶紧console一看,完全能用!!<br />
不太相信的关机,重新create了一次,还是没问题!<br />
把在/etc/xen下自动生成的test文件mv进/etc/xen/auto下,再把整个宿主机一重启,几分钟后重登陆一看,testOS也已经启动起来能用了。完成!<br />
<br />
</p>
perl的POD权限问题
2010-05-27T00:00:00+08:00
perl
http://chenlinux.com/2010/05/27/pod-problem-by-permission-of-tmp
<p>今天继续查找mod_perl对req_header的处理。</p>
<p>一开始打算用perldoc看Apache2::Request模块,结果在运行时出现如下错误:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Error in tempfile() using /tmp/XXXXXXXXXX:parent directory (./) is
not writable at /usr/lib/perl5/5.8.8/Pod/Perldoc.pm line 1483.
</code></pre>
</div>
<p>改到/tmp/执行命令,还是报错。看来和PWD是没关系,跟/tmp本身的权限有关吧~~(因为我经常在/tmp下做试验,可能不知道什么时候无意就改了权限了)</p>
<p>chmod 777 /tmp</p>
<p>再执行命令,ok了~~</p>
<hr />
<p>在看过Apache2::Request的doc后,没有发现header相关的设定,决定去直接看apache的那些pm,不过之前只管CPAN哗哗安装了,可从来没管过它们都安装在哪里……</p>
<p>/usr/五六个目录都是perl的,找起来可真不是个容易事~(记得之前测试,perl脚本每次执行,都有好几百毫秒用来查找模块在什么位置……)</p>
<p>一时偷懒去百度了一下,很不错,看到CPAN常见问题集,正好有这个办法:</p>
<p>perl -MFile::Find=find -MFile::Spec::Functions -Tlwe ‘find { wanted => sub { print canonpath $_ if /.pmz/ }, no_chdir => 1 }, @INC’</p>
<p>然后grep Apache,就看到结果了,都安装在/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/Apache2这个路径下。<br />
进去grep ‘$r->header’ *,立马就看出来,是RequestRec.pm里的。</p>
apache防盗链(mod_perl试用三)
2010-05-27T00:00:00+08:00
web
apache
perl
http://chenlinux.com/2010/05/27/anti_hotlinking-in-apache-by-mod_perl-3
<p>客户的要求,还剩下最后一步,就是referer限定。对于apache,有Mod_rewrite现成的可用:<br />
<code class="highlighter-rouge">apache
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^http://(www.)?test.com/.*$ [NC]
RewriteRule .mp3$ http://www.test.com/ [R=301,L]
</code><br />
不过既然之前已经用了perl,这里就一口气把perl写完吧:<br />
<code class="highlighter-rouge">perl
sub handler {
my $r = shift;
my $s = Apache2::ServerUtil->server;
my $secret = $r->dir_config('Secret') || '';#这里可以写成$r->dir_config->{Secret}
my $uri = $r->uri() || '';
my $expire = 2 * 3600;
+ my $referer = $r->headers_in->{Referer} || '';#这里却不可以写成$r->headers_in(Referer),会报错“argument is not a blessed reference (expecting an APR::Table derived object),不知道为什么?
+ if ($referer =~ m#^http://music.test.com#oi){
if ($uri =~ m#^/(d{4})(d{2})(d{2})(d{2})(d{2})/(w{32})(/S+.mp3)$#oi){
my ($year, $mon, $mday, $hour, $min, $md5, $path) = ($1, $2, $3, $4, $5, $6, $7);
my $str = md5_hex($secret . $year . $mon . $mday . $hour . $min . $path);
my $reqtime = mktime(00, $min, $hour, $mday, $mon - 1, $year - 1900);
my $now = time;
if ( $now - $reqtime < $expire){
if ($str eq $md5) {
$r->uri("$path");
return Apache2::Const::DECLINED;
}
}
}
+}
}
</code><br />
简单两句话,就ok了。测试如下:<br />
[27/May/2010:22:45:00 +0800] "GET /201005272218/ceaf967f6bcf9a185a3287b2e3ff5a02/smg/123.mp3 HTTP/1.0" 200 - <a href="http://music.test.com/">http://music.test.com</a> "Wget/1.10.2 (Red Hat modified)"<br />
[27/May/2010:22:45:05 +0800] "GET /201005272218/ceaf967f6bcf9a185a3287b2e3ff5a02/smg/123.mp3 HTTP/1.0" 404 - <a href="http://www.baidu.com/">http://www.baidu.com</a> "Wget/1.10.2 (Red Hat modified)"<br />
[27/May/2010:22:45:17 +0800] "GET /201005272218/ceaf967f6bcf9a185a3287b2e3ff5a03/smg/123.mp3 HTTP/1.0" 404 - <a href="http://music.test.com/">http://music.test.com</a> "Wget/1.10.2 (Red Hat modified)"<br />
对于各种非正常的访问,都返回404 NOT FOUND。<br />
如果想要返回403 ACCESS DENIED的话,经测试,在前面这些handler是没法做到的,必须在perlaccesshandler里才能return FORBIDDEN。那只能在之前的transhandler里统一改写uri成一个约定字符串,然后在access中再匹配拒绝。很麻烦。。。</p>
一个少见的squid报错
2010-05-27T00:00:00+08:00
squid
http://chenlinux.com/2010/05/27/a-non-host-error-in-squid
<p>今天有客户传过来一张报障截图,乍一看很正常的拒绝访问而已。可仔细一看,吓,地址栏里的url和errorpage返回的%U居然不一样!!<br />
<img src="/images/uploads/e68aa5e99a9ce688aae59bbe.jpg" /><br />
浏览器中写的是域名,squid却按配置拒绝的是对服务器ip发起的直接请求。<br />
赶紧去日志服务器上汇总deny信息,终于找到了相关日志。都是一个ip发过来的,大概如下:<br />
2010-05-25-14:04:13 0 60.209.232.219 TCP_MEM_HIT/200 15555 GET http://jobseeker.zhaopin.com/zhaopin/aboutus/law.html - NONE/- text/html “-“ “-“<br />
2010-05-25-14:05:19 2 60.209.232.219 TCP_DENIED/403 1464 GET http://113.6.255.97/zhaopin/aboutus/law.html - NONE/- text/html “-“ “-“<br />
2010-05-25-14:06:01 1 60.209.232.219 TCP_DENIED/403 1464 GET http://113.6.255.97/zhaopin/aboutus/law.html - NONE/- text/html “-“ “-“<br />
2010-05-25-14:10:00 1 60.209.232.219 TCP_DENIED/403 1464 GET http://113.6.255.97/zhaopin/aboutus/law.html - NONE/- text/html “-“ “-“<br />
2010-05-25-14:10:00 0 60.209.232.219 TCP_MEM_HIT/200 679 GET http://jobseeker.zhaopin.com/favicon.ico - NONE/- application/octet-stream “-“ “Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-TW; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 (.NET CLR 3.5.30729)”<br />
2010-05-25-14:12:39 2 60.209.232.219 TCP_DENIED/403 1464 GET http://113.6.255.97/zhaopin/aboutus/law.html - NONE/- text/html “-“ “-“<br />
2010-05-25-14:12:41 1 60.209.232.219 TCP_DENIED/403 1464 GET http://113.6.255.97/zhaopin/aboutus/law.html - NONE/- text/html “http://jobseeker.zhaopin.com/zhaopin/aboutus/law.html” “Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-TW; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 (.NET CLR 3.5.30729)”<br />
截图的那个时间点,日志中的user-agent居然是空!而且就在短短的一分钟前后,就连续出现正确和错误的反复访问。。。。。。在同事的提醒下,试着扫描了一下这个clientip,发现它还开着80端口:<br />
<code class="highlighter-rouge">bash
Starting Nmap 4.11 ( <a href="http://www.insecure.org/nmap/"><u><font color="#0000ff">http://www.insecure.org/nmap/</font></u></a> ) at 2010-05-25 17:22 CST
Interesting ports on 60.209.232.219:
Not shown: 1679 filtered ports
PORT STATE SERVICE
80/tcp open http
Nmap finished: 1 IP address (1 host up) scanned in 37.623 seconds
</code><br />
目前只能猜测这个ip应该是一个代理网关服务器,在转发访问请求的时候,把header中的Host信息给弄没了~~</p>
apache防盗链(modperl试用二)
2010-05-26T00:00:00+08:00
web
apache
perl
http://chenlinux.com/2010/05/26/anti_hotlinking-in-apache-by-mod_perl-2
<p>上回提到的防盗链方式是在strings上加上key和time,uri本身是不变的,这种方式其实现在不是很主流,主流的方式是将计算得出的加密串直接改在uri的路径里。比如下面将要提到的例子。要求其实和早先那个<a href="/2010/01/30/anti-hotlinking-in-nginx-lighttpd-squid">squid防盗链</a>的一模一样,就是改成用apache来跑。</p>
<p>Test.pm内容如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nb">package</span> <span class="nv">Test</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Socket</span> <span class="sx">qw(inet_aton)</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">POSIX</span> <span class="sx">qw(difftime mktime)</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Digest::</span><span class="nv">MD5</span> <span class="sx">qw(md5_hex)</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">RequestRec</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">Connection</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">RequestUtil</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">ServerUtil</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">Log</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">Request</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">Const</span> <span class="sx">qw(DECLINED FORBIDDEN)</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">handler</span> <span class="p">{</span>
<span class="err"> </span> <span class="k">my</span> <span class="nv">$r</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="err"> </span> <span class="k">my</span> <span class="nv">$s</span> <span class="o">=</span> <span class="nn">Apache2::</span><span class="nv">ServerUtil</span><span class="o">-></span><span class="nv">server</span><span class="p">;</span>
<span class="err"> </span> <span class="k">my</span> <span class="nv">$secret</span> <span class="o">=</span> <span class="nv">$r</span><span class="o">-></span><span class="nv">dir_config</span><span class="p">(</span><span class="s">'Secret'</span><span class="p">)</span> <span class="o">||</span> <span class="s">''</span><span class="p">;</span>
<span class="err"> </span> <span class="k">my</span> <span class="nv">$uri</span> <span class="o">=</span> <span class="nv">$r</span><span class="o">-></span><span class="nv">uri</span><span class="p">()</span> <span class="o">||</span> <span class="s">''</span><span class="p">;</span>
<span class="err"> </span> <span class="k">my</span> <span class="nv">$expire</span> <span class="o">=</span> <span class="mi">2</span> <span class="o">*</span> <span class="mi">3600</span><span class="p">;</span>
<span class="err"> </span> <span class="k">if</span> <span class="p">(</span><span class="nv">$uri</span> <span class="o">=~</span> <span class="nv">m</span><span class="c1">#^/(d{4})(d{2})(d{2})(d{2})(d{2})/(w{32})(/S+.mp3)$#oi) {</span>
<span class="err"> </span><span class="k">my</span> <span class="p">(</span><span class="nv">$year</span><span class="p">,</span> <span class="nv">$mon</span><span class="p">,</span> <span class="nv">$mday</span><span class="p">,</span> <span class="nv">$hour</span><span class="p">,</span> <span class="nv">$min</span><span class="p">,</span> <span class="nv">$md5</span><span class="p">,</span> <span class="nv">$path</span><span class="p">)</span> <span class="o">=</span> <span class="p">(</span><span class="nv">$1</span><span class="p">,</span> <span class="nv">$2</span><span class="p">,</span> <span class="nv">$3</span><span class="p">,</span> <span class="nv">$4</span><span class="p">,</span> <span class="nv">$5</span><span class="p">,</span> <span class="nv">$6</span><span class="p">,</span> <span class="nv">$7</span><span class="p">);</span>
<span class="err"> </span><span class="k">my</span> <span class="nv">$str</span> <span class="o">=</span> <span class="nv">md5_hex</span><span class="p">(</span><span class="nv">$secret</span> <span class="o">.</span> <span class="nv">$year</span> <span class="o">.</span> <span class="nv">$mon</span> <span class="o">.</span> <span class="nv">$mday</span> <span class="o">.</span> <span class="nv">$hour</span> <span class="o">.</span> <span class="nv">$min</span> <span class="o">.</span> <span class="nv">$path</span><span class="p">);</span>
<span class="err"> </span><span class="k">my</span> <span class="nv">$reqtime</span> <span class="o">=</span> <span class="nv">mktime</span><span class="p">(</span><span class="mo">00</span><span class="p">,</span> <span class="nv">$min</span><span class="p">,</span> <span class="nv">$hour</span><span class="p">,</span> <span class="nv">$mday</span><span class="p">,</span> <span class="nv">$mon</span> <span class="o">-</span> <span class="mi">1</span><span class="p">,</span> <span class="nv">$year</span> <span class="o">-</span> <span class="mi">1900</span><span class="p">);</span>
<span class="err"> </span><span class="k">my</span> <span class="nv">$now</span> <span class="o">=</span> <span class="nb">time</span><span class="p">;</span>
<span class="err"> </span><span class="k">if</span> <span class="p">(</span> <span class="nv">$now</span> <span class="o">-</span> <span class="nv">$reqtime</span> <span class="o"><</span> <span class="nv">$expire</span><span class="p">){</span>
<span class="err"> </span><span class="k">if</span> <span class="p">(</span><span class="nv">$str</span> <span class="ow">eq</span> <span class="nv">$md5</span><span class="p">)</span> <span class="p">{</span>
<span class="err"> </span> <span class="err"> </span><span class="nv">$r</span><span class="o">-></span><span class="nv">uri</span><span class="p">(</span><span class="s">"$path"</span><span class="p">);</span>
<span class="err"> </span> <span class="k">return</span> <span class="nn">Apache2::Const::</span><span class="nv">DECLINED</span><span class="p">;</span>
<span class="err"> </span><span class="p">}</span>
<span class="err"> </span><span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="mi">1</span><span class="p">;</span>
</code></pre>
</div>
<p>然后在httpd.conf中加上如下配置:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>PerlPostConfigRequire /home/apache2/perl/start.pl
SetHandler modperl
PerlTransHandler Test
PerlSetVar Secret abcdef
</code></pre>
</div>
<p>这里需要注意几点,根据modperl的处理流程,修改uri的时候,handler还没有走到对文件进行寻址,所以无法区分文件路径等信息,故而 <code class="highlighter-rouge">PerlTransHandler</code> 配置不能在 <code class="highlighter-rouge"><Directory></code> 和 <code class="highlighter-rouge"><Location></code> 里面。</p>
<p>而在<a href="/2010/04/15/anti_hotlinking-in-apache-by-mod_perl">试用一</a>里,核对strings是用的 <code class="highlighter-rouge">PerlAccessHandler</code>,当时已经确认了uri的文件路径,故而可以在 <code class="highlighter-rouge"><Location></code> 里。</p>
<p>另,上面的pm,对错误访问返回的是404,如果需要403,<code class="highlighter-rouge">return FORBIDDEN</code> 就可以了。</p>
<p>如果想同时根据referer来防盗链,可能要在 <code class="highlighter-rouge">PerlHeaderParserHandler</code> 阶段在进行一次判定了,这个还在研究,不知道怎么取request-header的信息……</p>
(读书笔记)网卡调优MTU
2010-05-23T00:00:00+08:00
linux
http://chenlinux.com/2010/05/23/learning-mtu
<p>还是那本《linux操作系统之奥秘》,系统管理性能调校章节。7.1.3 Jumbo Frame:</p>
<p>大概的意思,虽然现在的系统读写磁盘,内存、cpu等等都是好几K一块,而网卡则千兆万兆都有了;但有个基本的问题是,网卡在整整传输数据包的时候,其封包大小是另有限制的,即MTU(Maximum Transmission Unit,最大传输单元)。</p>
<p>书里打了个比方,以前人工搬运,一个纸箱子半米大刚好,现在用货车了,还是半米长的纸箱子,虽然一次装的确实是多了,不过密封拆封的次数和耗时还是没少的……完全可以换成集装箱嘛~~这就是jumbo frame。</p>
<p>当然,这个集装箱目前还有很多问题,第一、他还不是一个标准化的东东;第二、他不能单方面做决定(你用集装箱运过去了,收局却只会拆纸箱~);第三、丢帧的冗错更复杂了(这个书里没讲,不过我想应该会吧~)……肯定还有别的问题,不然标准化组织肯定早解决了。</p>
<p>由这些情况来看,在internet上大规模运用,肯定是不行的。不过在服务器集群里头的SAN上,收发端都是自己掌控,又不用担心跨网的丢包,或许还是不错的运用。</p>
<hr />
<p>今天又看到扶凯一篇nfs优化。在[rw]size上做了些文章,这是nfs服务器和客户端的传输块大小。顿时想到之前看到的MTU了。如果把MTU调整一下,不刚好可以配合一下了?</p>
<p>结果百度了一下,果然已经有修改MTU的nfs优化文章了。先用ping -s和nfsstat -o net确认不同封包情况下的丢包率,然后ifconfig eth0 mtu修改MTU值。</p>
<p>这里自然可以运用之前我写过的netperf了,呵呵~<br />
打算有时间,试试去。</p>
bash使用技巧
2010-05-23T00:00:00+08:00
bash
http://chenlinux.com/2010/05/23/intro-bash-usage
<p>几个跟ctrl有关的操作,很有爱,贴一下:</p>
<ol>
<li>最常见的:^l,相当于clear命令清屏;</li>
<li>最有用的:^r,自动打开一个(reverse-i-search)`’: 的提示符,可以搜索之前的history,注意:搜索内容仅限于command,不包括option和file。不过我试验对于wget等命令,又可以搜索整行任意内容……汗</li>
<li>
<p>常用的一个系列:</p>
<p>^a,回到行首,写完一串命令后突然发现最前头要修改一下~<br />
^e,回到行尾,其实这两个最常用的情况我是在写for循环执行的时候;<br />
^u,剪切从行首到当前光标的所有内容;<br />
^k,剪切从当前光标到行尾的所有内容;<br />
^y,粘贴刚才剪切的内容——在辛辛苦苦敲好一长串命令,却发现要先执行别的命令才行的时候,这三条操作就很有用了。<br />
^w,删除从当前光标到向左最近一个空格之前的内容;<br />
^t,交换当前光标和光标左边字符的位置,这个一般不怎么用,就一个字符嘛,直接重写好了~~<br />
^p,执行上一条命令。</p>
</li>
</ol>
<hr />
<p>几个关于history的操作和配置,同样有爱,继续粘贴:<br />
1. 最常用的,看操作时间和扩容:HISTTIMEFORMAT=’%F %T ‘和HISTFILESIZE=2000、HISTSIZE=2000;<br />
2. 可能很有用的,HISTCONTROL=ignoredups或者erasedups甚至ignorespace:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>ignoredups连续执行的命令只记录一次,比方在做wget测试的时候,可能一口气就敲了十多次~~
erasedups哪怕不是连着执行的,也只记录一次;
ignorespace如果在命令前加个空格,这条命令就不记录进history了~
</code></pre>
</div>
<ol>
<li>更狠一点,有些太常用的命令懒得加空格,也不想记录,直接HISTIGNORE=”pwd:ls”</li>
<li>最狠的,history -c全删了~~</li>
</ol>
<hr />
<ol>
<li>执行最近一次类似的命令:比如!tai</li>
<li>执行history里的第100条命令:!100</li>
<li>上一条命令:!!,另外的衍生物!!:n即上一条命令的第几个参数,^为第一个,$为最后一个。这两个情况可以简写成!^和!$</li>
<li>推论,最近一次类似命令的参数!tai:n</li>
</ol>
mod_perl处理流程图一张~
2010-05-22T00:00:00+08:00
web
perl
apache
http://chenlinux.com/2010/05/22/intro-mod_perl-process-by-image
<p>在<a href="http://www.fayland.org/journal">http://www.fayland.org/journal</a>上看到一张图,感觉很舒服很明白,转帖过来。<br />
<img alt="" src="http://www.fayland.org/journal/img/http_cycle_all.gif" /></p>
<hr />
<p>其实mod_perl官网有也有这个类似的图,不过没这个pp~~</p>
<p>总的来说,mod_perl不同的指令,就是分别插入到这个圆圈上的标签位置,完成不同的作用。</p>
<p>比如说,PerlTransHandler可以做url_rewrite;PerlHeaderParserHandler可以判断request_header;PerlAccessHandler可以做访问控制;PerlAuthenHandler可以做用户验证;PerlLogHandler可以记录日志。</p>
<p>如果perl程序return的是Apache2::Const::OK,那就直接结束这个圆环,进入之后apache该干的事情去;如果是Apache2::Const::DECLINED,那就表示这个handler正常完成了,继续沿着圆环走吧~</p>
<p> 针对每个handler的具体可配置,还是看官网吧:<a href="http://perl.apache.org/docs/2.0/user/handlers/http.html" target="_blank">http://perl.apache.org/docs/2.0/user/handlers/http.html</a></p>
<p>之前我就是这个地方犯错了,在httpd.conf里指定的accesshandler,程序却想完成rewrite的功能,结果一直报404……</p>
strace进程跟踪排错一例
2010-05-21T00:00:00+08:00
linux
strace
http://chenlinux.com/2010/05/21/intro-strace
<p>在对HTTPS进行反向代理的时候,如果源站未能提供SSL的cert和key,可以采用TCP协议的端口转发完成。最常见的是iptables,还有rinetd。之前的博文中都有提到。 <br />
今天碰到一个事情,使用rinetd进行443转发的客户,全网都无法访问了…… <br />
proxy上对源站443端口能telnet通,绑定ie访问源站是没有问题;proxy的443端口本地也能telnet通,但是ie访问就是打不开任何页面。 <br />
偶然想起strace,觉得对rinetd进程进行跟踪试试。strace -p 13916,然后这边打开ie,输入https的域名,回车…… <br />
看到服务器tty上显示如下信息: <br />
select(16, [4], [], NULL, NULL) = 1 (in [4]) <br />
#rinetd程序处于select(),运行的FD为4 <br />
accept(4, {sa_family=AF_INET, sin_port=htons(3766), sin_addr=inet_addr(“211.99.216.18”)}, [16]) = 6 <br />
#从我本机的3766端口发起请求,被服务器接受,FD为6 <br />
ioctl(6, FIONBIO, [1]) = 0 <br />
#设定该socket为阻塞状态 <br />
setsockopt(6, SOL_SOCKET, SO_LINGER, [0], 4) = -1 EINVAL (Invalid argument) <br />
#设定其为异步处理方式,不错失败了……汗 <br />
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 7 <br />
#打开FD为7的socket,tcp传输方式为stream流方式 <br />
bind(7, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr(“0.0.0.0”)}, 16) = 0 <br />
#服务器打开对任一端口的监听 <br />
setsockopt(7, SOL_SOCKET, SO_LINGER, [0], 4) = -1 EINVAL (Invalid argument) <br />
ioctl(7, FIONBIO, [1]) = 0 <br />
#设定这个向源请求的socket为已阻塞 <br />
connect(7, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr(“222.73.34.25”)}, 16) = -1 EINPROGRESS (Operation now in progress) <br />
#向222.73.34.25的443端口发送请求 <br />
select(16, [4 6 7], [], NULL, NULL) = 1 (in [6]) <br />
recvfrom(6, “2631A1=31K36536423”217:352346225207N7qP’342!2008”…, 1024, 0, NULL, NULL) = 70 <br />
#从FD6即连接客户电脑的socket收到的内容 <br />
select(16, [4 6 7], [7], NULL, NULL) = 2 (in [7], out [7]) <br />
recvfrom(7, 0x111e6860, 1024, 0, 0, 0) = -1 EHOSTUNREACH (No route to host) <br />
#从FD7即回源的socket收到的内容——“无法找到连接主机的路由”!! <br />
close(7) = 0 <br />
#关闭FD7 <br />
select(16, [4 6], [6], NULL, NULL) = 1 (out [6]) <br />
time(NULL) = 1274410006 <br />
stat(“/etc/localtime”, {st_mode=S_IFREG|0644, st_size=405, …}) = 0 <br />
close(6) = 0 <br />
#关闭FD6,这个是我关闭ie后关闭的。 <br />
很奇怪呀,这个222.73.34.25是什么地址?ping客户域名返回的明明不是这个ip呀? <br />
试着kill rinetd,然后重启动rinetd服务。再重复如上操作。相关内容如下: <br />
accept(4, {sa_family=AF_INET, sin_port=htons(3917), sin_addr=inet_addr(“211.99.216.18”)}, [16]) = 6 <br />
ioctl(6, FIONBIO, [1]) = 0 <br />
setsockopt(6, SOL_SOCKET, SO_LINGER, [0], 4) = -1 EINVAL (Invalid argument) <br />
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 7 <br />
bind(7, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr(“0.0.0.0”)}, 16) = 0 <br />
setsockopt(7, SOL_SOCKET, SO_LINGER, [0], 4) = -1 EINVAL (Invalid argument) <br />
ioctl(7, FIONBIO, [1]) = 0 <br />
connect(7, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr(“219.235.4.17”)}, 16) = -1 EINPROGRESS (Operation now in progress) <br />
#回源地址是219.235.4.17,这个正是ping出来的正确源ip! <br />
select(18, [4 6 7], [], NULL, NULL) = 1 (in [6]) <br />
recvfrom(6, “2631A1=31K365365f334370210367202260B33332E3337232423340s21”…, 1024, 0, NULL, NULL) = 70 <br />
select(18, [4 6 7], [7], NULL, NULL) = 1 (out [7]) <br />
sendto(7, “2631A1=31K365365f334370210367202260B33332E3337232423340s21”…, 70, 0, NULL, 0) = 70 <br />
select(18, [4 6 7], [], NULL, NULL) = 1 (in [7]) <br />
recvfrom(7, “2631r2432F31K365365U10200u236332*Mf316l2252306v254350364”…, 1024, 0, NULL, NULL) = 1024 <br />
select(18, [4 6], [6], NULL, NULL) = 1 (out [6]) <br />
…… <br />
#这些recv和send就是页面请求的传输过程,ie上显示出来正确的页面了。<br />
返回去查找之前的工单,也确认了222.73.34.25正是该客户之前的源站ip,而后改成219.235.4.17的。<br />
由此看来rinetd虽然可以在conf里写域名,但其对域名的解析,只会在启动的时候执行一次,之后就一直持续对固定的那个ip进行转发了!</p>
<hr />
<table>
<tbody>
<tr>
<td>马后炮的说:这个问题其实被我搞复杂了。因为连接是stream的方式,有一个keepalive的时间,所以完全可以在ie访问的时候用netstat -an</td>
<td>grep :443,就能看到回源的ip,很轻松的就能判定源站ip有问题了——不过这是现在按图索骥,在不知道是源ip不对之前,谁会想到呢~~~</td>
</tr>
</tbody>
</table>
url_rewrite配置的小区别
2010-05-20T00:00:00+08:00
squid
http://chenlinux.com/2010/05/20/little-diff-in-url_rewrite
<p>一直以为squid的url_rewrite就是改写url后,传给squid分析是否缓存,然后返回缓存或者回源。在浏览器地址栏上的url是不变的。<br />
今天才知道在print $uri的时候,可以给他加上http_code。变成print 302:$uri的格式,然后就可以由浏览器发起302跳转到新页面了。</p>
服务器登陆欢迎信息~
2010-05-18T00:00:00+08:00
linux
http://chenlinux.com/2010/05/18/intro-motd
<p>上同事的测试机,发现登陆的时候在显示PS1前还显了一行’Welcome to Cloudex’的欢迎信息,蛮好玩的。<br />
于是去百度一下,原来是设置/etc/issue和/etc/motd文件就可以了。打开/etc/issue,里面已经有两行centos5的信息,先加这里试试,保存退出,重新ssh上服务器,结果还是默认的:<br />
Connecting to 192.168.0.1:22…<br />
Connection established.<br />
Escape character is <a href="mailto:'^@]'">’^@]’</a>.<br />
Last login: Tue May 18 16:04:38 2010 from 192.168.1.1<br />
[root@test ~]#<br />
再仔细看看,似乎这个文件得restart后才能生效。<br />
再去修改motd,退出重登陆。还是不行……这就怪了~~~<br />
这事儿过身就忘了,直到今天,看/etc/ssh/sshd_config,正好看到里面有一条PrintMotd no,莫非就是这个!?赶紧改成PrintMotd yes,/etc/init.d/sshd restart;exit,重登陆。果然看到之前卸载motd里的欢迎信息了~<br />
Last login: Tue May 18 16:04:38 2010 from 192.168.1.2<br />
haha,I’m Raocl!<br />
[root@sdl4 ~ 16:04:46]#<br />
OK啦~~</p>
(读书笔记)pushd/popd命令
2010-05-14T00:00:00+08:00
linux
http://chenlinux.com/2010/05/14/intro-pushd-popd
<p>买了一本《linux操作系统之奥秘》,话说本人对海峡那边的书一向是抱有一定的认可的~ <br />
出于好奇,先翻看了第七章“系统性能”——也是最短的一章。其中提到CPU节能与性能的关联、管理和观察(这段拗口否~) <br />
于是也上自己测试机去看cpufreq。结果发现/proc下没有相关路径,也就是说没用节能,然后又去找cpuspeed,在/etc/init.d/下找到了cpuspeed的启动脚本。不过启用是才发现有些模块在内核中没有编译,即使用modprobe也加载不上…… <br />
于是在init.d下瞎逛,看看系统shell脚本,算是学习吧~ <br />
还真看到一个新奇玩意儿了:pushd/popd命令。 <br />
在shell里操作的时候,经常在不同路径之间切换,最经常的办法:cd -;最简单的办法:上下键翻history;最安全的办法:啥都用全路径。 <br />
而pushd命令,则是创建一个堆栈,专门用来存储路径位置的。使用很简单, <br />
pushd 路径1 <br />
就自动到了“路径1”下,同时把“路径1”从左向右压入堆栈; <br />
等压完一堆以后,如何切换路径呢? <br />
堆栈从左到右(或者说成从下向上?)分别是num 0,1,2…… <br />
要切换到第几个路径,pushd +n即可——注意此时pushd自动把该路径变成last了。 <br />
然后还有popd命令,从堆栈中弹出路径,这个命令可以不接options,默认从last弹出(堆栈的LIFO原则);也可以和pushd一样用+n的方式指定弹出哪个;比较搞怪的是,如果在popd后面随便跟一个路径,不管这个路径是不是堆栈中存在的,popd都会从右边把first路径弹出……汗~~</p>
关闭snmp和nrpe的syslog正常输出
2010-05-11T00:00:00+08:00
linux
snmp
nagios
syslog
http://chenlinux.com/2010/05/11/stop-snmp-nrpe-output-into-syslog
<p>默认安装启动的snmp,会把日志记录在系统日志/var/log/messages里。</p>
<p>特别郁闷的一点是,哪怕一次snmp连接请求,它也要记录上好几句。。。系统日志呀,多少关键信息,就这样湮没在snmp的刷屏里了……</p>
<p>于是决定关掉这些输出。找了找,snmp.conf里好像没有关于log-file的配置,ps看进程,/usr/sbin/snmpd后面跟了长长一大串的options,于是觉得可以看看–help和/etc/init.d/里的启动脚本。</p>
<p>果然看到-L参数,而进程中启动的真是Ls:facility: log to syslog (via the specified facility)!!</p>
<p>指定这部分option(即Ls)为LS 2,就可以了!</p>
<p>而/etc/init.d/snmpd中,这部分定义如下:<br />
<code class="highlighter-rouge">bash
if [ -e /etc/snmp/snmpd.options ]; then
. /etc/snmp/snmpd.options
else
OPTIONS="-Lsd -Lf /dev/null -p /var/run/snmpd.pid -a"
fi
</code><br />
可见/etc/snmp/snmpd.options优先级比OPTIONS高,而且修改单独文件也比修改系统启动脚本放心些。<br />
<code class="highlighter-rouge">bash
cat > /etc/snmp/snmpd.options <<EOF
OPTIONS="-LS 2 d -Lf /dev/null -p /var/run/snmpd.pid -a"
EOF
</code><br />
/etc/init.d/snmpd restart,等等再看,messages里果然没有snmp的刷屏了~~<br />
(注:不同版本的OS启动脚本不同,请自行参考。至少我手头的服务器上就还有在/etc/sysconfig/snmpd.options里的,且须写成Ls而不是LS)</p>
<hr />
<p>然后是nrpe,这也是个刷屏的高手,而且处理起来比snmp还麻烦。因为在nrpe.cfg中,同样没有关于log_file的记录,也没有单独的启动options,因为它是交给xinetd去管理的。</p>
<p>xinetd呀,听着就让我动手的胆子小了不少~~小心着看吧</p>
<p>xinetd –help,呃,压根没输出;</p>
<p>/etc/init.d/xinetd,除了LANG基本没什么定义……</p>
<p>再进/etc/xinetd.d/,赫然发现一个文件叫nrpe!赶紧打开一看,有一条“log_on_failure += USERID”,改成“log_on_failure =”,保存退出,重启观察,messages里还是出现新nrpe的条目,失败了……</p>
<p>等等,之前为什么是+=呢?返回上层目录,看/etc/xinetd.conf,原来有“Define general logging characteristics”,默认的日志记录选项,包括log_on_failure/log_on_success和log_type。嗯,看来就是log_type了!</p>
<p>vi /etc/xinetd.d/nrpe,插入log_type SYSLOG daemon info……当然是不行的。百度一下xinetd文档,原来这里有七个等级,看样子应该是从高到低:emerg,alert,crit,err,warning,notice,info,debug。我就改个warning看看吧,不行!alert,还不行!!</p>
<p>发怒了,用emerg——电脑铛铛作响,原来emerg不单记录syslog,还强制显示在tty上…(估计是因为nrpe记录都是start/stop,算起来都是最高级别的动作吧)…就xinetd这速度,都搞的没法干活了~~摸索着捣鼓回原状,继续研究吧~</p>
<p>log_type除了syslog,还有file soft hard方式 ,单独记录。那我输出到空不就行了?于是写成log_type = /dev/null,重启报错:“wrong number of arguments [file=/etc/xinetd.d/nrpe] [line=13]”。文档明确说了soft和hard是可以在不写的情况下默认为5MB 5.05MB的,那只能是file写的格式不对了。</p>
<p>改成log_type = file /dev/null,重启。等呀等呀,五分钟过去了,messages里还是没有nrpe的记录,成功了!</p>
perl脚本性能优化(续)
2010-05-11T00:00:00+08:00
perl
http://chenlinux.com/2010/05/11/performance-optimization-of-perl-script-2
<p>上回提到,性能优化的四个回答,今天在扶凯的博客里看到一篇文章,刚好是同样情况下的流程优化。按照其中的说法,修改测试,果然改进很大:</p>
<p>优化的地方在这里:</p>
<p>-:foreach (sort keys %url_table) {<br />
+:if(exists $url_table{$1}) {<br />
print LIST_FH “$_n”;}</p>
<p>在将所有的url写入散列后,一般的处理办法是用keys检索哈希表,而其实只需要判定存在,直接输出即可。<br />
然后分别用time测试四个脚本,结果如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>长正则,检索哈希——1.882s
短正则,检索哈希——1.543s
长正则,判定输出——0.804s
短正则,判定输出——0.651s
</code></pre>
</div>
perl边学边练(purge脚本进阶)
2010-05-07T00:00:00+08:00
CDN
squid
perl
http://chenlinux.com/2010/05/07/purge-cache-by-perl-script-2
<p>之前的purge脚本usage是./purge.pl “url1” “url2”。如果url变成成百上千个那么多的时候,这样就不行了。也需要在脚本中处理成文件句柄。修改如下:<br />
```perl<br />
#!/usr/bin/perl -w<br />
use IO::Socket;<br />
unless (@ARGV == 2) { die “usage: $0 ip.list url.list” }<br />
open(HOST,”<$ARGV[0]”)||die “cannot open the ip list”;<br />
open(URL,”<$ARGV[1]”)||die “cannot open the url list”;<br />
$EOL = “15121512”;</p>
<p>while (defined($ip=<host>)){
seek URL, 0, 0;
while (defined($uri=<url>)){
$remote = IO::Socket::INET->new( Proto => "tcp",
PeerAddr => $ip,
PeerPort => "80",
);
unless ($remote) { die "cannot connect to http daemon on $ip" }
$remote->autoflush(1);
print $remote "PURGE $uri HTTP/1.0".$EOL;
while ( <$remote> ) { print }
close $remote;
}
}
close(URL);
close(HOST);
```
在改编时碰到的问题,或者说学习到的东西就是这个while双重嵌套里句柄的问题。
一开始,我写成了 `open;open;while(a){while(b){}};close;close;` 这样子。结果输出结果只能执行完b循环就正常退出了。
然后修改成 `open;open;while(a){while(b){}close;};close;` 这样,结果在执行完一遍b循环后,继续的a循环提示打开的句柄已关闭——这证明a循环是执行了的,只是没结果……
再修改成 `open;while(a){open;while(b){}close;};close;` 这样,结果执行结果正常了!然后想到这么频繁的打开关闭句柄或许效率不太好,于是继续寻找办法。
最后修改成这样 `open;open;while(a){seek B,0,0;while(b){}};close;close;` 即上面代码段的样子,执行结果也正常。
关键在这个 `seek` 函数,用途是在(文件较大的情况下)指定从哪个位置开始读取:
usage:seek FILEHANDLE文件句柄,POSITION 读取开始位置字节,WHENCE(0-重新开始、1-当前开始、2-文件最后开始)
因为在第一遍b循环的时候,文件句柄已经读取到了文件最后,所以在下一次循环的时候,要用seek,0,0返回文件初始位置,重新逐行输入。
而之前的脚本因为while里嵌套的是foreach,已经一次性将文件读入数组了,所以不存在句柄的问题。</url></host></p>
用Devel::NYTProf模块排查优化perl脚本性能
2010-05-07T00:00:00+08:00
perl
http://chenlinux.com/2010/05/07/performance-optimization-of-perl-script-by-devel-nytprof
<p>缓存服务器上有一个perl写的日志分析脚本,记录所有不重复的url。之后对squid进行目录刷新时,从记录下来的文件中查找匹配的url即可。</p>
<p>不过这些天服务器老是出现负载报警,用top观察,这个 <code class="highlighter-rouge">url_parser.pl</code> 脚本一旦执行时,就占用了高达90%的CPU和40%的MEM。wc看存储的url.list文件,有大概4,000,000行;<code class="highlighter-rouge">url.$(date).list</code> 当前有140,000行。</p>
<p>于是上CU去请教perl执行效率的查找思路。</p>
<p>回复有:1、正则精准度;2、文件读取效率;3、全局变量数;4、频繁打开句柄;5、流程优化</p>
<p>比如读取文件不要用 <code class="highlighter-rouge">@line=FILE</code> 用 <code class="highlighter-rouge">while(<FILE>)</code> ;正则<code class="highlighter-rouge">^</code> 句首,带上 <code class="highlighter-rouge">/oi</code> ;注意哈希表与内存交换区等等;最后推荐给我 <code class="highlighter-rouge">Devel::NYTProf</code> ,进行测试。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nv">perl</span> <span class="o">-</span><span class="nv">MCPAN</span> <span class="o">-</span><span class="nv">e</span> <span class="nv">shell</span>
<span class="o">></span><span class="nv">install</span> <span class="nn">JSON::</span><span class="nv">Any</span><span class="err">(不安这个东东,在</span><span class="nv">nyt</span><span class="err">生成</span><span class="nv">html</span><span class="err">的时候会报</span><span class="nv">warning</span><span class="err">,不过不安也可以)</span>
<span class="o">></span><span class="nv">install</span> <span class="nn">Devel::</span><span class="nv">NYTProf</span>
</code></pre>
</div>
<p>然后采用 <code class="highlighter-rouge">perl -d:NYTProf /home/purge/url_parser.pl</code> 运行脚本,会在当前路径下生成nytprof.out。</p>
<p>再用 <code class="highlighter-rouge">nytprofhtml nytprof.out</code> 生成web页面。</p>
<p>另开一个apache,将生成的nytprof目录发布出来。用ie打开即可看到了,如下:</p>
<p><img src="/images/uploads/nytprof-index.jpg" alt="nytprof" /></p>
<p>下面还有载入模块的时间。之前我用strace跟踪了一下脚本的运行,发现在载入pm的时候,perl会搜索好多乱七八糟的目录,最后才正确,还一度担心是因为这个原因浪费了时间和资源呢。不过根据测试结果来看,载入模块总共花了不到30ms,不是什么可怕的事情。</p>
<p>然后点击 <code class="highlighter-rouge">/home/purge/url_parser.pl</code> 的 <code class="highlighter-rouge">reports(line·block·sub)</code>,可以看到具体每个语句的执行情况:</p>
<p><img src="/images/uploads/devel-nytprof-time.jpg" alt="nytprof-0" /></p>
<p>打开十四万行的url文件花了2.14s,然后再用2.09s将它们载入哈希表中;</p>
<p><img src="/images/uploads/devel-nytprof-time1.jpg" alt="nytprof-1" /></p>
<p>打开正在运行的access.log(5分钟截取一次,squidclient mgr:5min里rps为17.65,即大概该有5000行以下;结果显示是3306 calls)并截取其中的url,花了141ms,然后再用42.6ms载入哈希表中;</p>
<p><img src="/images/uploads/devel-nytprof-time2.jpg" alt="nytprof-2" /></p>
<p>最后,用919ms对哈希表排序,用1.58s重记录整个url文件。</p>
<p>(143677-143579=98,即3306条日志中有98条是新增url)</p>
<p>注意到第二张图中,对access.log分析时,match那步,<strong>每行花了30us</strong>!而在对urllist和tmplog分析时,每行只花3-4us的样子。看来是这一步的正则写的不好了,如下:<br />
<code class="highlighter-rouge">perl
my $log_pattern = qr '^.*?d+s+w+s+(http://.+?)s+.+';
</code></p>
<p>根据日志的格式和需求,改成这样 <code class="highlighter-rouge">my $log_pattern = qr 's(http://.+?)s';</code> 其他不变,再次测试,该部分的测试结果如下:</p>
<p><img src="/images/uploads/devel-nytprof-time3.jpg" alt="nytprof-3" /></p>
<p>__降低成7us每行__啦!效果明显呀~~</p>
web服务监控小工具httping
2010-05-06T00:00:00+08:00
monitor
http://chenlinux.com/2010/05/06/intro-httping
<p>今天偶然看到这个工具,感觉挺有用的,记录一下。<br />
安装过程很简单:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget http://www.vanheusden.com/httping/httping-1.4.1.tgz
tar zxvf httping-1.4.1.tgz -C /tmp/
<span class="nb">cd</span> /tmp/httping-1.4.1/
make <span class="o">&&</span> make install
</code></pre>
</div>
<p>默认就安装在/usr下了。如果不想,直接改Makefile去。<br />
然后使用:</p>
<p>httping -options</p>
<h3 id="g-url">-g url</h3>
<p>### -h hostname<br />
### -p port<br />
### -x host:port(如果是测squid,用-x,不要用-h;和curl的不一样,curl -H指定的是发送的hostname,这个-h是指定给DNS解析的hostname)<br />
### -c count<br />
### -t timeout<br />
### -s statuscode<br />
### -S 将时间分开成连接和传输两部分显示<br />
### -G GET(默认是HEAD)<br />
### -b 在使用了GET的前提下显示传输速度KB/s<br />
### -B 同-b,不过使用了压缩<br />
### -I useragent<br />
### -R referer<br />
### -C cookie=*<br />
### -l SSL<br />
### -U username<br />
### -P password<br />
### -n a,b 提供给nagios监控用的,当平均响应时间>=a时,返回1;>=b,返回2;默认为0<br />
### -N c 提供给nagios监控用的,一切正常返回0,否则只要有失败的就返回c</p>
<p>举例如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>httping -x 211.151.78.37:80 http://bj.qu114.com/ -SGbs -c 10
Using proxyserver: 211.151.78.37:80
PING bj.qu114.com:80 (http://bj.qu114.com/):
connected to bj.qu114.com:80, seq=0 time=27.00+2945.88=2972.87 ms 200 OK 16KB/s
connected to bj.qu114.com:80, seq=1 time=27.09+2233.38=2260.47 ms 200 OK 17KB/s
connected to bj.qu114.com:80, seq=2 time=26.90+168.70=195.60 ms 200 OK 400KB/s
connected to bj.qu114.com:80, seq=3 time=26.89+2524.52=2551.41 ms 200 OK 15KB/s
connected to bj.qu114.com:80, seq=4 time=26.90+1939.48=1966.37 ms 200 OK 20KB/s
connected to bj.qu114.com:80, seq=5 time=26.79+2085.52=2112.31 ms 200 OK 18KB/s
connected to bj.qu114.com:80, seq=6 time=27.04+1294.78=1321.82 ms 200 OK 32KB/s
connected to bj.qu114.com:80, seq=7 time=26.97+2527.29=2554.26 ms 200 OK 15KB/s
connected to bj.qu114.com:80, seq=8 time=26.88+1498.28=1525.16 ms 200 OK 27KB/s
connected to bj.qu114.com:80, seq=9 time=27.21+1208.70=1235.91 ms 200 OK 34KB/s
--- http://bj.qu114.com/ ping statistics ---
10 connects, 10 ok, 0.00% failed
round-trip min/avg/max = 195.6/1869.6/2972.9 ms
Transfer speed: min/avg/max = 15/59/400 KB
</code></pre>
</div>
netperf网络测试
2010-05-05T00:00:00+08:00
linux
http://chenlinux.com/2010/05/05/intro-netperf
<p>本文的主要参考是IBM工作室的一篇文章:<a href="http://www.ibm.com/developerworks/cn/linux/l-netperf/index.html" target="_blank">http://www.ibm.com/developerworks/cn/linux/l-netperf/index.html</a></p>
<p>文中指出网络性能的五个衡量指标,前两个是可用性和响应时间,这也是最经常关注的,因为有最常见的ping命令;然后是利用率、可用带宽和剩余带宽。</p>
<ul>
<li>
<p>安装:<br />
<code class="highlighter-rouge">bash
wget ftp://ftp.netperf.org/netperf/netperf-2.4.5.tar.gz
tar zxvf netperf-2.4.5.tar.gz -C /tmp
cd /tmp/netperf-2.4.5
./configure && make && make install
</code><br />
然后在服务器端运行/usr/local/bin/netserver启动服务,可以看到如下正确结果:</p>
<p>Starting netserver at port 12865<br />
Starting netserver at hostname 0.0.0.0 port 12865 and family AF_UNSPEC</p>
</li>
</ul>
<p>同样在客户端(即网络另一头的服务器上)安装好,就可以用/usr/local/bin/netperf工具进行测试了。</p>
<ul>
<li>我的测试环境是:</li>
</ul>
<p>S:121.14.225.197 汕头电信 <br />
C:218.60.36.39 沈阳网通</p>
<p>绝对的跨网测试了,呵呵~</p>
<p>一、先测试默认的TCP_STREAM批量传输,结果如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>/usr/local/bin/netperf -H 121.14.225.197 -l 60
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 121.14.225.197 (121.14.225.197) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
873800 163840 163840 60.46 6.52
</code></pre>
</div>
<p>即电信设备采用873800字节的socket接收缓存,网通设备采用163840字节的socket发送缓存,测试60.46秒后,网络吞吐量为6.52Mb/s。</p>
<p>逐次采用-m修改发送包的大小后,发现在如下情况时,网络吞吐量是最好的(是默认发送分组的两倍半):</p>
<div class="highlighter-rouge"><pre class="highlight"><code>/usr/local/bin/netperf -H 121.14.225.197 -l 60 -- -m 8192
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 121.14.225.197 (121.14.225.197) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
873800 163840 8192 61.46 17.60
</code></pre>
</div>
<p>可见在南北互联中间某环节的路由器,限定了socket缓冲区的大小。</p>
<p>二、然后测试UDP的批量:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>/usr/local/bin/netperf -t UDP_STREAM -H 121.14.225.197 -l 60 -- -m 8192
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 121.14.225.197 (121.14.225.197) port 0 AF_INET
Socket Message Elapsed Messages
Size Size Time Okay Errors Throughput
bytes bytes secs # # 10^6bits/sec
6553600 8192 60.01 6973545 0 7615.92
6553600 60.01 393669 429.93
</code></pre>
</div>
<p>可以看到,只有5.65%的UDP分组被接收了……</p>
<p>三、小请求大应答模式的测试</p>
<p>1、数据库请求(一次TCP连接,反复传输数据):</p>
<div class="highlighter-rouge"><pre class="highlight"><code>/usr/local/bin/netperf -t TCP_RR -H 121.14.225.197 -- -r 32,1024
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 121.14.225.197 (121.14.225.197) port 0 AF_INET
Local /Remote
Socket Size Request Resp. Elapsed Trans.
Send Recv Size Size Time Rate
bytes Bytes bytes bytes secs. per sec
163840 873800 32 1024 10.00 18.49
163840 873800
</code></pre>
</div>
<p>在32字节请求和1024字节响应的情况下,平均交易率为18.49次/秒。</p>
<p>2、HTTP请求(每次传输都新建连接):</p>
<div class="highlighter-rouge"><pre class="highlight"><code>/usr/local/bin/netperf -t TCP_CRR -H 121.14.225.197 -- -r 32,1024
TCP Connect/Request/Response TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 121.14.225.197 (121.14.225.197) port 0 AF_INET
Local /Remote
Socket Size Request Resp. Elapsed Trans.
Send Recv Size Size Time Rate
bytes Bytes bytes bytes secs. per sec
163840 873800 32 1024 10.00 9.30
163840 873800
</code></pre>
</div>
<p>RPS几乎下降一般,可见TCP建连是很耗时间的。</p>
<p>3、还可以测试UDP_RR,理论上来说,去除掉TCP建连的耗时,UDP_RR的RPS应该有所提高。不过在我的跨网环境下,rps几乎低到0.1,考虑到之前测试得到的5%的收发率,这个测试基本只能在局域网内才有意义。</p>
系统消息队列(squid启动小故障)
2010-04-30T00:00:00+08:00
linux
squid
http://chenlinux.com/2010/04/30/ipcs-error-in-squid-start
<p>昨天公司一台服务器被机房动手动脚之后,上面的虚拟机变得极不正常。squid基本跑不了三四个小时就out of memory一次。虽然已经把cache_mem调低到free的1/4了。依然如此。<br />
更过分的事情刚才发生了,在又一次挂掉后,squid重启动彻底失败起不来了。<br />
赶紧查看cache.log,其中记载了失败的原因,如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>storeDiskdInit: msgget: (28) No space left on device
FATAL: msgget failed
</code></pre>
</div>
<p>一眼看起来像是磁盘空间满了,df一看,没问题呀,use才1%呢!<br />
然后注意到其具体失败处,是msgget函数。msgget是用来新建/获取信息队列的。也就是说,信息队列的某个方面满了。查看一下这方面的信息:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[root@sitesquid1 ~]# ipcs -l
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 67108864
max total shared memory (kbytes) = 17179869184
min seg size (bytes) = 1
------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767
------ Messages: Limits --------
max queues system wide = 16
max size of message (bytes) = 65536
default max size of queue (bytes) = 65536
</code></pre>
</div>
<p>看着有点晕,队列系统宽度的最大值,这个比较难理解(下面明明有最大大小了呀?)。只好求助百度,原来就是能打开的消息队列的最大个数,或者说消息队列标识符的最大值。<br />
看这个值挺小的,再看网上也有DB2和apache修改这个值的说法,那就修改一下试试:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sysctl -w kernel.msgmni=128
</code></pre>
</div>
<p>再启动squid,OK!!<br />
再重新看看消息队列标识符到底打开了多少:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[root@spshort5 ~]# ipcs|awk '/msqid/{a=NR}END{print NR-a}'
19
</code></pre>
</div>
<p>果然是超过16。</p>
awk取数值小技巧
2010-04-29T00:00:00+08:00
bash
awk
http://chenlinux.com/2010/04/29/tech-to-get-int-in-awk
<p>今天在Q群里看到有人在取ping值时用的小技巧,很是不错,加深了对awk的理解。<br />
ping命令输出如下:</p>
<p>64 bytes from xd-22-5-a8.bta.net.cn (202.108.22.5): icmp_seq=1 ttl=55 time=20.7 ms</p>
<p>要取20.7出来,一般会指定=和” “两个FS,然后取NF-1列。</p>
<p>不过这个群友给出另一个办法,指定=为FS,然后取$NF+0,其值就是20.7了!</p>
<p>很好很好,采用一个+0的计算,等于是指定前面的$NF为数字型,于是~~</p>
<p>不过这个字母不能在数字前面,否则awk会认为是0。例如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>echo 123ms|awk '{print $1+0}'
123
echo ms123|awk '{print $1+0}'
0
</code></pre>
</div>
at命令
2010-04-26T00:00:00+08:00
bash
http://chenlinux.com/2010/04/26/intro-at-command
<p>经常使用crontab做定时任务。不过偶然碰到只需要半夜执行一次就够了的时候,还用crontab的话,第二天还得记得上去删除掉任务。就比较麻烦了——尤其是我记忆力不太好~~<br />
好在发现了at命令:</p>
<p>首先启动at服务/etc/init.d/atd start</p>
<p>然后at -f test.sh -v 17:10</p>
<p>系统返回</p>
<div class="highlighter-rouge"><pre class="highlight"><code>job 1 at 2010-04-27 17:10
</code></pre>
</div>
<p>然后at -f ptest.sh 2:00 july 11<br />
系统返回</p>
<div class="highlighter-rouge"><pre class="highlight"><code>job 2 at 2010-07-11 02:00
</code></pre>
</div>
<p>用atq查看任务队列</p>
<div class="highlighter-rouge"><pre class="highlight"><code>2 2010-07-11 02:00 a root
1 2010-04-27 16:10 a root
</code></pre>
</div>
<p>用at -c [job号]查看任务内容</p>
<p>用atrm [job号]删除任务</p>
shell并发脚本学习
2010-04-25T00:00:00+08:00
bash
http://chenlinux.com/2010/04/25/learning-concurrent-shell-script
<p>在CU上看到的老帖子,创建并发程序的shell。个人觉得非常经典,贴回来好好学习使用。用()包围的是我写的学习笔记,#的是原帖注释:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c">#!/usr/bin/ksh(自然我得把这里改成bash)</span>
<span class="c"># SCRIPT: ptest.sh</span>
<span class="c"># AUTHOR: Ray001(呃,这些也是要学习滴,版权意识嘛~)</span>
<span class="c"># DATE: 2008/10/03</span>
<span class="c"># REV: 2.0</span>
<span class="c"># For STUDY</span>
<span class="c"># PURPOSE:</span>
<span class="c"># 实现进程并发,提高执行效率,同时能记录每个执行失败的子进程信息</span>
<span class="c">#定义并发进程数量</span>
<span class="nv">PARALLEL</span><span class="o">=</span>3
<span class="c">#定义临时管道文件名(我为这个定义想了好久,不知道$$.是什么特殊变量;后来实际上测试机一实验才发现把事情想复杂了……这就是以“脚本pid+.fifo”组成的字符串而已。完全可以改写成其他样子。</span>
<span class="nv">TMPFILE</span><span class="o">=</span><span class="nv">$$</span>.fifo
<span class="c">#定义导出配置文件全路径名(其实我个人很诧异为什么这里要定义到家目录去,这样在touch ptest.cfg的时候多麻烦呀)</span>
<span class="nv">CMD_CFG</span><span class="o">=</span><span class="nv">$HOME</span>/cfg/ptest.cfg
<span class="c">#定义失败标识文件</span>
<span class="nv">FAILURE_FLAG</span><span class="o">=</span>failure.log
<span class="c">####################### 函数定义 ########################</span>
<span class="c"># 中断时kill子进程(学习重点一:kill -9 0——杀死脚本自己及衍生出来的子进程,嗯,全家自杀)</span>
<span class="k">function </span>trap_exit
<span class="o">{</span>
<span class="nb">kill</span> -9 0
<span class="o">}</span>
<span class="c"># 通用执行函数</span>
exec_cmd<span class="o">()</span>
<span class="o">{</span>
<span class="c"># 此处为实际需要执行的命令,本例中用sleep做示例</span>
sleep <span class="k">${</span><span class="nv">1</span><span class="k">}</span>
<span class="k">if</span> <span class="o">[</span> <span class="nv">$?</span> -ne 0 <span class="o">]</span>
<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"命令执行失败"</span>
<span class="k">return </span>1
<span class="k">fi</span>
<span class="o">}</span>
<span class="c">####################### 主程序 ########################</span>
<span class="c">#(学习重点二:当信号为1、2、3、15时,执行''中的命令,即调用trap_exit函数自杀,然后退出该shell并返回信号2——把0、1留给后面用)</span>
<span class="nb">trap</span> <span class="s1">'trap_exit; exit 2'</span> 1 2 3 15
<span class="c">#清理失败标识文件</span>
rm -f <span class="k">${</span><span class="nv">FAILURE_FLAG</span><span class="k">}</span>
<span class="c">#为并发进程创建相应个数的占位(其实这个定义不绝对正确有效,比如占了十个位,但cfg中只有7个参数传递,这个脚本就只有7个并发,不过这是小节,无关大雅~)</span>
<span class="c">#(创建命名管道)</span>
mkfifo <span class="nv">$TMPFILE</span>
<span class="c">#(学习重点三:为命名管道指定文件标识符为4,<>分别是输入和输出,即绑定了该管道的输入输出都在4这个文件标识符上!)</span>
<span class="nb">exec </span>4<><span class="nv">$TMPFILE</span>
<span class="c">#(删除管道文件,不知道这步是为什么,目前只能猜测是担心程序运行时该文件被其他人或者程序误用吧?)</span>
rm -f <span class="nv">$TMPFILE</span>
<span class="c">#(用{}和用()的区别在shell是否会衍生子进程。let命令用以变量运算。这一段给文件标识符输入了几个回车。)</span>
<span class="c">#(不知道这几个回车和“并发占位”什么关系。找了很久,没发现??目前猜测是管道的流处理所以每个子进程用一行)</span>
<span class="o">{</span>
<span class="nv">count</span><span class="o">=</span><span class="nv">$PARALLEL</span>
<span class="k">while</span> <span class="o">[</span> <span class="nv">$count</span> -gt 0 <span class="o">]</span>
<span class="k">do
</span><span class="nb">echo
let </span><span class="nv">count</span><span class="o">=</span><span class="nv">$count</span>-1
<span class="k">done</span>
<span class="o">}</span> >& 4
<span class="c">#从任务列表 seq 中按次序获取每一个任务</span>
<span class="c">#(从家目录下那个cfg文件中读取sleep的时间)</span>
<span class="k">while </span><span class="nb">read </span>SEC
<span class="k">do</span>
<span class="c">#(从标识符中读取回车?不懂,还是管道和子进程的问题……)</span>
<span class="nb">read</span> <& 4
<span class="c">#(后台执行主程序命令或者输出错误日志,完成后清空标识符)</span>
<span class="o">(</span> exec_cmd <span class="k">${</span><span class="nv">SEC</span><span class="k">}</span> <span class="o">||</span> <span class="nb">echo</span> <span class="k">${</span><span class="nv">SEC</span><span class="k">}</span>>><span class="k">${</span><span class="nv">FAILURE_FLAG</span><span class="k">}</span> ; <span class="nb">echo</span> >&amp;4 <span class="o">)</span> &
<span class="k">done</span><<span class="nv">$CMD_CFG</span>
<span class="c">#(等待子进程结果返回值)</span>
<span class="nb">wait</span>
<span class="c">#(关闭文件标识符4)</span>
<span class="nb">exec </span>4>&-
<span class="c">#并发进程结束后判断是否全部成功</span>
<span class="k">if</span> <span class="o">[</span> -f <span class="k">${</span><span class="nv">FAILURE_FLAG</span><span class="k">}</span> <span class="o">]</span>
<span class="k">then
</span><span class="nb">exit </span>1
<span class="k">else
</span><span class="nb">exit </span>0
<span class="k">fi</span>
</code></pre>
</div>
起点小说网的cdn分析~(绝非正式报告)
2010-04-23T00:00:00+08:00
CDN
http://chenlinux.com/2010/04/23/cdn-analysis-for-www-qidian-com
<p>每天习惯了在起点小说网看看小说轻松一下神经,今天一不小心,看见了squid错误页面。于是小小的查看了一下起点的cdn状况。<br />
首先声明本身的接入情况:宽带通4M接入,ip138显示为北京网通adsl……<br />
访问的是决战朝鲜的第三十三章。页面内容来看,域名主要有www.qidian.com、image.cmfu.com、ipagent.igalive.com、cj.qidian.com四个,前两个在网宿加速(lxdns)、第三个在蓝汛加速(ccgslb)、第四个在起点母公司盛大加速(sdo),其余零散域名未加速。<br />
lxdns返回的ip是天津网通(ping值7ms);<br />
ccgslb返回的ip是北京蓝汛(ping值8ms);<br />
sdo返回的是上海联通(本地ping不通,在沈阳网通测试机上居然ping通了,返回的是同一个ip,30ms,汗)……<br />
整个页面一共81个对象:<br />
加速的域名uedas.qidian.com和sdo的cj.qidian.com下的对象,time都在100ms以上;<br />
网宿加速域名下的对象,主要是小图片和页面;<br />
图片小到以B计算,返回time在16ms左右;页面包括aspx和html,html大小在15-20KB左右,返回time在50ms左右;<br />
耗时最大的是主页面http://www.qidian.com/BookReader/1501306,27478560.aspx,耗时437ms,其中15ms建立链接,而422ms传输内容,content-length 69408,请求头带有no-cache的cache-control和gzip,deflate的压缩;返回头带有Age 1的HIT结果,但X-Cache有两层,分别是一层MISS一层HIT,via头信息是jsyz232:80 (Cdn Cache Server V2.0), tg146:80 (Cdn Cache Server V2.0)。预计应该是parent上设定强制缓存,而leaf上不缓存;<br />
另一个耗时较大的是http://www.qidian.com/Javascript/ReadChapterNew.js?t=091216,耗时250ms,content-length 35423,其他情况和aspx差不多,不过返回了三个X-Cache,一个MISS两个HIT,在via头上很奇怪的看到三个服务器分别是jsyz232:80 (Cdn Cache Server V2.0), zb99:80 (Cdn Cache Server V2.0), tg134:8103 (Cdn Cache Server V2.0),不知道这个开8103端口的是什么意思??(网上查了一下,民生网银用的是这个端口,~~)<br />
最后是蓝汛加速域名下的对象,都是广告;<br />
js文件url是相同的,都是http://ipagent.igalive.com/show_ads.js,但先后请求了5次,返回时间依次为:4.28s、48.42s、3.14s、6.17s和875ms,波动相当大!观察这5次的Age,分别是778、827、782、834、835;对应返回时间,可以看出是同一台服务器返回的同一份缓存。再具体看time的细分,建立连接时间都很短,传输内容时间从500ms到1.5s不等。也就是说,主要波动在于服务器的响应时间上。<br />
aspx?文件,url比较长,类似这种:http://ipagent.igalive.com/s.aspx?adid=100144&host=http://ipagent.igalive.com&dt=1272030174514&lmt=1272028966&output=101&url=http://www.qidian.com/BookReader/1501306/27478560.aspx&ref=http://me.qidian.com/BookCase/1/1&flash=10&u_h=660&u_w=1126&u_ah=627&u_aw=1126&u_cd=24&u_tz=480&u_his=1&u_nplug=7&u_nmime=16,都是MISS回源的。长度和上面的js差不多都在40KB左右。但是到最后甚至有一个文件连接超时了……</</p>
<hr />
<p>综上:<br />
网宿给起点的cache服务器,应该是针对超小文件做过优化的,所以在16KB以下的图片,返回TCP_MEM_HIT;超过这个大小的那两个文件,响应速度就下降的比较厉害。<br />
蓝汛给起点的cache服务器,负载压力可能比较大,导致响应速度很慢且有极大波动;我猜测对于js文件,其服务器可能还采用了304回源;从回源的aspx来看,回源效果很差很差……<br />
盛大给起点的cache服务器,不说啥了~~<br />
最后附squid的错误信息ERR_READ_TIMEOUT图:<br />
<img src="/images/uploads/err_read_timeout.jpg" alt="" /></p>
perl内置变量
2010-04-22T00:00:00+08:00
perl
http://chenlinux.com/2010/04/22/perl-built_in-variables
<p>$_ 默认输入/模式搜索空间,常用于-f -d测试、print unlink函数、m// s/// tr///匹配、foreach while循环</p>
<p>@_ 传递给函数的所有参数</p>
<p>$&/$<code class="highlighter-rouge">/$' 分别是上次匹配成功(时/前/后)的字符串(这三个变量会导致效率显著降低)
local $_ = 'abcdefghi';
/def/;
print "$</code>:$&:$’n”; # prints abc:def:ghi</p>
<p>$+ 上次搜索中最后一个括号匹配的文本<br />
/Version: (.<em>)|Revision: (.</em>)/ && ($rev = $+);</p>
<p>$^N 上次搜索中最后闭合的组所匹配的文本(可能嵌套)<br />
(?:(…)(?{ $var = $^N })); #这样可以省去计算括号个数</p>
<p>$* 匹配模式优化(即当其==0时,perl认为字符串仅为一行(n),推荐使用/s和/m代替此变量)</p>
<p>$. 相当于awk中的NR,但如果文件句柄被close(),$.就会复位,相当于FNR</p>
<p>$/ 相当于RS,正常情况设置为字符串;设置为undef即一口气全读入;设置为整数标量时则读入小于该整数的记录而不是行;一些小文件,可以采用$data = do { local $/;<fh>; };单行命令读入,就是这个原理。</fh></p>
<p>$ 相当于ORS</p>
<p>$, 相当于OFS</p>
<p>$” 同$,不过用于向用”引起的字符串插入数据的时候</p>
<p>$; 模拟多维数组时下标的分隔符。<br />
$foo{$a,$b,$c};<br />
$foo{join($;, $a, $b, $c)}; #两者相当</p>
<table>
<tbody>
<tr>
<td>$</td>
<td>设置为非零值时,强制刷新每次输出缓存;在向管道和套接字输出时设置该变量</td>
</tr>
</tbody>
</table>
<p>$# 相当于OFMT,不过awk里是%.6g,perl里是%<em>n</em>g,<em>n</em>取决于OS上float.h 中 DBL_DIG 宏的值(不建议使用该变量)</p>
<p>@-/@+ 分别是模式匹配时,每次成功匹配字符串开始和结束处的偏移量</p>
<p>在对某个变量 $var 进行匹配后:<br />
$` 和 “substr($var, 0, $-[0])” 相同<br />
$& 和 “substr($var, $-[0], $+[0] - $-[0])” 相同<br />
$’ 和 “substr($var, $+[0])” 相同<br />
$1 和 “substr($var, $-[1], $+[1] - $-[1])” 相同<br />
$2 和 “substr($var, $-[2], $+[2] - $-[2])” 相同</p>
<p>$#+是最近成功匹配了多少个组</p>
<p>$[ 数组中的第一个元素/字符串中第一个字符的索引号,默认为0(最好不要改)</p>
<p>$] perl解释器版本(5.001格式)</p>
<p>$^V perl解释器版本(5.8.8格式)</p>
<p>$^T 程序运行时间(UNIX秒)</p>
<p>$$ perl脚本的运行进程号</p>
<p>$< perl脚本的运行实际用户</p>
<p>$> perl脚本的运行有效用户</p>
<p>$( perl脚本的运行实际用户组</p>
<p>$) perl脚本的运行有效用户组</p>
<p>$0 当前程序名</p>
<p>$@/$!/$^E/$? 分别是perl解释器、C库、操作系统、外部程序检测到的错误<br />
<code class="highlighter-rouge">perl
eval q{
open my $pipe, "/cdrom/install |" or die $!;
my @res = <$pipe>;
close $pipe or die "bad pipe: $?, $!";
};
</code><br />
在需要 “eval” 的字符串没有通过编译(若 “open” 或 “close”导入的原型错误则可能发生)或者 Perl 代码在执行过程中 die() 掉,则 $@变量会被设置。这些情况下 $@ 的值是编译错误信息或 “die” 的参数(其中会内插 $! 和 $?)。(另见 Fatal) <br />
上面的 eval() 表达式执行后,open()、”<pipe>" 和 "close" 被翻译成对 C运行库的调用,继而 进入操作系统内核。若其中某个调用失败,则 $! 会设置为C 库的 "errno" 值。</pipe></p>
<p>在少数操作系统下,$^E 可能含有更详细的错误指示,例如“CDROM仓门没有关闭”。不支持扩展错误 信息的系统只是将 $^E 设置为和 $!一样的值。</p>
<p>最后,$? 在外部程序 /cdrom/install 失败时设置为非 0 值。高8位反映出该程序遇到的特定错误 条件(程序的 exit() 值),低8位反映失败方式,例如信号致死或核心转储,细节参见 wait(2)。对比仅在检测到错误条件时才设置的 $! 和 $^E,变量 $? 在每个 “wait” 或管道”close” 时都会 设置并冲掉旧值。这一行为更接近 $@,后者在每次 eval()后总是在失败时设置并在成功时清除。</p>
<p>更多细节请分别参见 $@、$!、$^E 和 $? 各自的说明。</p>
要命的刷新
2010-04-22T00:00:00+08:00
CDN
http://chenlinux.com/2010/04/22/nginx-default-proxy_cache_key
<p>今天一天都在跟刷新做斗争。<br />
先是squid的目录刷新,用/home/squid/bin/squidclient -p 80 mgr:objects|awk ‘/harbin/{system(“/home/squid/bin/squidclient -p 80 -m purge “$2}’刷新一遍;客户反馈看到的依然是旧页面;我想想似乎看到url里有带()的,awk的system函数这时候会出错;于是改成用for i in <code class="highlighter-rouge">/home/squid/bin/squidclient -p 80 mgr:objects|awk '/harbin/{print $2}'</code>;do /home/squid/bin/squidclient -p 80 -m purge “$i”;done再刷新一遍;然后自己绑定节点访问,居然也还看到是旧页面!</p>
<p>通过httpwatch抓取,其中的http://www.harbin-beer.cn/flash/age.swf的Age高达23325,显然没有被purge到,单独提交/home/squid/bin/squidclient -p 80 -m purge http://www.harbin-beer.cn/flash/age.swf,再访问,Ok了!!<br />
毫无疑问http://www.harbin-beer.cn/flash/age.swf这个url绝对是能被/harbin/模式匹配的,但为什么之前刷新不到?而且不单单是某一台服务器如此。。。</p>
<p>然后是nginx的url刷新,某小图片加速客户原先是增量缓存,于是nginx中只是很简单的配置了一下文件类型和缓存时间。不料今天客户突然传过来一个24M大小的url列表,将近30万条url要求全网刷新!而这批nginx连后台刷新接口都没有……哭<br />
临时更换nginx版本为–add-ngx_cache_purge的。在设置proxy_cache_purge时却又碰到了难题。因为之前的cache配置里压根没配置proxy_cache_key!!有心格盘,但一算,300000*5k=1.5G,而cache已存文件是100G,格盘动作太大了……<br />
进到nginx的cache目录下,strings其中的文件,看到如下信息:<br />
<code class="highlighter-rouge">bash
[root@ct5 ~]# strings /cache/0/00/c7de957045a9987b18f94d3cc1f99000 |head
KEY: http://images6.anjukestatic.com/property/20090904/22/65/91/69/22659169/600x600.jpg
HTTP/1.0 200 OK
Server: nginx
Date: Thu, 11 Mar 2010 15:48:20 GMT
Content-Type: image/jpeg
ajk: server=img01-001
Expires: Thu, 31 Dec 2037 23:55:55 GMT
Cache-Control: max-age=315360000
X-Cache: HIT from CDN01-001
Age: 1465699
</code><br />
对照其他带purge的nginx_cache格式,可以发现nginx默认的proxy_cache_key应该是$scheme://$host$uri$is_args$args,那么proxy_cache_purge就设成$scheme://$host$1$is_args$args,重读配置,然后curl -x 127.0.0.1:80 http://images6.anjukestatic.com/purge/property/20090904/22/65/91/69/22659169/600x600.jpg,看到<br />
```html</p>
<html>
<head><title>Successful purge</title></head>
<body bgcolor="white">
<center><h1>Successful purge</h1>
<br />Key : http://images6.anjukestatic.com/property/20090904/22/65/91/69/22659169/600x600.jpg
<br />Path: /cache/0/00/c7de957045a9987b18f94d3cc1f99000
</center>
<hr /><center>nginx/0.7.65</center>
</body>
</html>
<p>```<br />
(写到这里,想到google首页源代码省略了</body></html>,据说是因为就少这几个字符,在全球就能节省几个G的带宽。相比来说这个purgemodule可真浪费的)<br />
再ls /cache/0/00/c7de957045a9987b18f94d3cc1f99000,提示No such file or directory,成功了。接下来就是for循环刷新了……</p>
sscanf用法
2010-04-20T00:00:00+08:00
C
http://chenlinux.com/2010/04/20/sscanf-usage
<p>int sscanf(</p>
<p>const char *buffer,</p>
<p>const char *format [,argument ] …</p>
<p>);<br />
最简单的举例:切割时间</p>
<p>char sztime1[16] = “”, sztime2[16] = “”;<br />
sscanf(“2006:03:18 - 2006:04:18”, “%s - %s”, sztime1, sztime2);</p>
<p>可是如果时间是”2006:03:18-2006:04:18”,即没有空格时,%s的定义就没法用了。这时候可以使用%[..]来定义,如下</p>
<p>sscanf(“2006:03:18-2006:04:18”, “%[0-9,:]-%[0-9,:]”, sztime1, sztime2);</p>
<p>%[]的用法,类似正则表达式,可以采用[a-z]这样的匹配;可以采用[^a-z]这样的排除匹配;还可以采用*[a-z]这样的匹配过滤。举例如下:</p>
<p>const char sourceStr[] = “hello, world”;</p>
<p>char buf[10] = {0};</p>
<p>sscanf(sourceStr, “%*s%s”, buf);</p>
<p>执行结果就是过滤了hello,打印出world~~</p>
iscsi试验(读写测试)
2010-04-15T00:00:00+08:00
linux
http://chenlinux.com/2010/04/15/intro-iscsi-2
<p>隔了一天,回头再来mount /dev/sda /mnt;成功了~<br />
不信邪,再上一台,login,mount,继续成功……<br />
难道是前天rp不行?</p>
<p>然后试验写入。发现在每台client上的写入,都暂时不会显示在server和其他client上。只有经过logout和login后才会显示。<br />
接着进行一下简单的读写速度测试,先是read,用hdparm工具,结果如下:</p>
<p>服务器端:[root@ct5 ~]# hdparm -tT /dev/xvdb1<br />
/dev/xvdb1:<br />
Timing cached reads: 20880 MB in 1.99 seconds = 10486.02 MB/sec<br />
Timing buffered disk reads: 162 MB in 3.01 seconds = 53.85 MB/sec</p>
<p>客户端:[root@ct5 ~]# hdparm -tT /dev/sda<br />
/dev/sda:<br />
Timing cached reads: 21944 MB in 1.99 seconds = 11022.51 MB/sec<br />
Timing buffered disk reads: 152 MB in 3.02 seconds = 50.41 MB/sec</p>
<p>然后用apache发布,wget的速度基本也差不多。</p>
<p>然后是write,用time dd,最开始指定了bs=4k,结果让我很惊讶</p>
<p>服务器端:[root@ct5 cache]# time dd if=/dev/zero of=rao bs=4k count=655360<br />
655360+0 records in<br />
655360+0 records out<br />
2684354560 bytes (2.7 GB) copied, 35.1299 seconds, 76.4 MB/s <br />
real 0m35.204s <br />
user 0m0.440s <br />
sys 0m6.968s</p>
<p>而客户端2684354560 bytes (2.7 GB) copied, 28.584 seconds, 93.9 MB/s <br />
real 0m28.650s <br />
user 0m0.496s <br />
sys 0m7.404s</p>
<p>居然比服务器端还快。于是想到iscsi的封包机制,可能bs4k有影响,于是往下降成2k,结果立刻有体现了</p>
<p>服务器1342177280 bytes (1.3 GB) copied, 6.93954 seconds, 193 MB/s <br />
real 0m6.943s <br />
user 0m0.428s <br />
sys 0m4.848s</p>
<p>客户端1342177280 bytes (1.3 GB) copied, 12.5357 seconds, 107 MB/s <br />
real 0m12.539s <br />
user 0m0.348s <br />
sys 0m4.720s</p>
<p>再降成1k,结果稍慢</p>
<p>服务器671088640 bytes (671 MB) copied, 4.1199 seconds, 163 MB/s <br />
real 0m4.123s <br />
user 0m0.372s <br />
sys 0m3.280s</p>
<p>客户端671088640 bytes (671 MB) copied, 7.06874 seconds, 94.9 MB/s <br />
real 0m7.093s <br />
user 0m0.328s <br />
sys 0m3.324s <br />
可见在写速率上,还是有一定差距的。</p>
apache防盗链(modperl试用)
2010-04-15T00:00:00+08:00
web
apache
perl
http://chenlinux.com/2010/04/15/anti_hotlinking-in-apache-by-mod_perl
<p>客户需求如下:</p>
<p>在web请求视频时,按算法生成密文和明文串,然后依规则组成最终的url请求;</p>
<p>算法规则:</p>
<p>用如下三个关键词生成 MD5 密文:</p>
<ol>
<li>自定义密钥:abcde.;</li>
<li>视频文件真实路径,即/path/to/file.rmvb;</li>
<li>请求时间,以当前UNIX时间换算为十六进制字符串,并作为明文;</li>
</ol>
<p>最终url格式是 <code class="highlighter-rouge">http://www.test.com/path/to/file.rmvb?key=1234567890abcdefghijklmnopqrstuy&t=1234abcd</code> 这样。</p>
<p>要求失效时间为8小时。</p>
<p>这个需求和之前一次相当类似,不过上回是squid,这次是apache。同样采用perl脚本进行防盗链设置,apache需要使用modperl模块。</p>
<p>首先安装perl模块:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget http://perl.apache.org/dist/mod_perl-2.0-current.tar.gz
tar zxvf mod_perl-2.0-current.tar.gz
<span class="nb">cd </span>mod_perl-2.0-current.tar.gz
perl Makefile.PL <span class="nv">MP_APXS</span><span class="o">=</span>/home/apache2/bin/apxs
make <span class="o">&&</span> make install
<span class="nb">echo</span> <span class="s2">"LoadModule perl_module modules/mod_perl.so"</span> >> /home/apache2/conf/httpd.conf
perl -MCPAN -e shell
>install Apache2::Request
>look Apache2::Request
rm -f configure
rm -f apreq2-config
./buildconf
perl Makefile.PL
make <span class="o">&&</span> make install
<span class="nb">exit</span>
<span class="c">#因为64位系统的libexpat.so有问题,编译libapreq2会出问题,只好如此强制安装</span>
<span class="nb">echo</span> <span class="s2">"LoadModule apreq_module modules/mod_apreq2.so"</span> >> /home/apache2/conf/httpd.conf
</code></pre>
</div>
<p>因为libapreq2.so安装在/home/apache2/lib/下了,所以需要<code class="highlighter-rouge">echo "/home/apache2/lib" >/etc/lo.so.conf.d/apache.conf</code>,然后ldconfig。</p>
<p>修改httpd.conf,加入如下设置:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>PerlPostConfigRequire /home/apache2/perl/start.pl
<Location /smg>
SetHandler modperl
PerlAccessHandler DLAuth
PerlSetVar ShareKey abcde.
</Location>
</code></pre>
</div>
<p>然后mkdir /home/apache2/perl/,在其中创建start.pl和DLAuth.pm两个文件。start.pl文件内容如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">lib</span> <span class="sx">qw(/home/apache2/perl)</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">RequestIO</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">RequestRec</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">Connection</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">RequestUtil</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">ServerUtil</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">Log</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">Request</span> <span class="p">();</span>
<span class="mi">1</span><span class="p">;</span>
</code></pre>
</div>
<p>DLAuth.pm文件内容如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nb">package</span> <span class="nv">DLAuth</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">Socket</span> <span class="sx">qw(inet_aton)</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">POSIX</span> <span class="sx">qw(difftime strftime)</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Digest::</span><span class="nv">MD5</span> <span class="sx">qw(md5_hex)</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">RequestIO</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">RequestRec</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">Connection</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">RequestUtil</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">ServerUtil</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">Log</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">Request</span> <span class="p">();</span>
<span class="k">use</span> <span class="nn">Apache2::</span><span class="nv">Const</span> <span class="o">-</span><span class="nv">compile</span> <span class="o">=></span> <span class="sx">qw(OK FORBIDDEN)</span><span class="p">;</span>
<span class="k">sub </span><span class="nf">handler</span> <span class="p">{</span>
<span class="err"> </span> <span class="k">my</span> <span class="nv">$r</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="err"> </span> <span class="k">my</span> <span class="nv">$s</span> <span class="o">=</span> <span class="nn">Apache2::</span><span class="nv">ServerUtil</span><span class="o">-></span><span class="nv">server</span><span class="p">;</span>
<span class="err"> </span> <span class="k">my</span> <span class="nv">$shareKey</span> <span class="o">=</span> <span class="nv">$r</span><span class="o">-></span><span class="nv">dir_config</span><span class="p">(</span><span class="s">'ShareKey'</span><span class="p">)</span> <span class="o">||</span> <span class="s">''</span><span class="p">;</span>
<span class="err"> </span> <span class="k">my</span> <span class="nv">$uri</span> <span class="o">=</span> <span class="nv">$r</span><span class="o">-></span><span class="nv">uri</span><span class="p">()</span> <span class="o">||</span> <span class="s">''</span><span class="p">;</span>
<span class="err"> </span> <span class="k">my</span> <span class="nv">$args</span> <span class="o">=</span> <span class="nv">$r</span><span class="o">-></span><span class="nv">args</span><span class="p">()</span> <span class="o">||</span> <span class="s">''</span><span class="p">;</span>
<span class="err"> </span> <span class="k">my</span> <span class="nv">$expire</span> <span class="o">=</span> <span class="mi">8</span> <span class="o">*</span> <span class="mi">3600</span><span class="p">;</span>
<span class="err"> </span> <span class="k">if</span> <span class="p">(</span><span class="nv">$args</span> <span class="o">=~</span> <span class="nv">m</span><span class="c1">#^key=(\w{32})&t=(\w{8})$#i) {</span>
<span class="err"> </span> <span class="k">my</span> <span class="p">(</span><span class="nv">$key</span><span class="p">,</span> <span class="nv">$date</span><span class="p">)</span> <span class="o">=</span> <span class="p">(</span><span class="nv">$1</span><span class="p">,</span> <span class="nv">$2</span><span class="p">);</span>
<span class="err"> </span> <span class="k">my</span> <span class="nv">$str</span> <span class="o">=</span> <span class="nv">md5_hex</span><span class="p">(</span><span class="nv">$shareKey</span> <span class="o">.</span> <span class="nv">$uri</span> <span class="o">.</span> <span class="nv">$date</span><span class="p">)</span>
<span class="err"> </span> <span class="k">my</span> <span class="nv">$reqtime</span> <span class="o">=</span> <span class="nb">hex</span><span class="p">(</span><span class="nv">$date</span><span class="p">);</span>
<span class="err"> </span> <span class="k">my</span> <span class="nv">$now</span> <span class="o">=</span> <span class="nb">time</span><span class="p">;</span>
<span class="err"> </span> <span class="k">if</span> <span class="p">(</span> <span class="nv">$now</span> <span class="o">-</span> <span class="nv">$reqtime</span> <span class="o"><</span> <span class="nv">$expire</span><span class="p">)</span> <span class="p">{</span>
<span class="err"> </span> <span class="err"> </span> <span class="k">if</span> <span class="p">(</span><span class="nv">$str</span> <span class="ow">eq</span> <span class="nv">$key</span><span class="p">)</span> <span class="p">{</span>
<span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="k">return</span> <span class="nn">Apache2::Const::</span><span class="nv">OK</span><span class="p">;</span>
<span class="err"> </span> <span class="err"> </span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="nv">$s</span><span class="o">-></span><span class="nv">log_error</span><span class="p">(</span><span class="s">"[$uri FORBIDDEN] Auth failed"</span><span class="p">);</span>
<span class="err"> </span> <span class="err"> </span> <span class="err"> </span> <span class="k">return</span> <span class="nn">Apache2::Const::</span><span class="nv">FORBIDDEN</span><span class="p">;</span>
<span class="err"> </span> <span class="err"> </span> <span class="p">}</span>
<span class="err"> </span> <span class="p">}</span>
<span class="err"> </span> <span class="p">}</span>
<span class="err"> </span> <span class="nv">$s</span><span class="o">-></span><span class="nv">log_error</span><span class="p">(</span><span class="s">"[$uri FORBIDDEN] Auth failed"</span><span class="p">);</span>
<span class="err"> </span> <span class="k">return</span> <span class="nn">Apache2::Const::</span><span class="nv">FORBIDDEN</span><span class="p">;</span>
<span class="p">}</span>
<span class="mi">1</span><span class="p">;</span>
</code></pre>
</div>
<p>就可以了。</p>
<p>apachectl restart。测试一下,先用perl自己生成一个测试链接:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/perl -w</span>
<span class="k">use</span> <span class="nn">Digest::</span><span class="nv">MD5</span> <span class="sx">qw(md5_hex)</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$key</span> <span class="o">=</span> <span class="s">"bestv."</span><span class="p">;</span>
<span class="nv">$path</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">(</span><span class="nv">@ARGV</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$date</span> <span class="o">=</span> <span class="nb">sprintf</span><span class="p">(</span><span class="s">"%x"</span><span class="p">,</span><span class="nb">time</span><span class="p">);</span>
<span class="nv">$result</span> <span class="o">=</span> <span class="nv">md5_hex</span><span class="p">(</span><span class="nv">$key</span> <span class="o">.</span> <span class="nv">$path</span> <span class="o">.</span> <span class="nv">$date</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$uri</span> <span class="o">=</span> <span class="s">"http://127.0.0.1$path?key=$result&t=$date"</span><span class="p">;</span>
<span class="k">print</span> <span class="nv">$uri</span><span class="p">;</span>
</code></pre>
</div>
<p>运行 <code class="highlighter-rouge">./url.pl /smg/abc.rmvb</code> 生成 <code class="highlighter-rouge">http://127.0.0.1/smg/abc.rmvb?key=4fb6b4e6a0ec484aea98fa727fc7149d&t=4bc7dd5a</code>,然后 <code class="highlighter-rouge">wget -S -O /dev/null "http://127.0.0.1/smg/abc.rmvb?key=4fb6b4e6a0ec484aea98fa727fc7149d&t=4bc7dd5a"</code>,返回 200 OK;任意修改 t 为 12345678,再 wget,返回 403 Forbidden。<code class="highlighter-rouge">error_log</code> 显示如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[Fri Apr 16 11:47:06 2010] [error] [/smg/abc.rmvbkey=4fb6b4e6a0ec484aea98fa727fc7149d&t=12345678 FORBIDDEN] Auth failed
</code></pre>
</div>
iscsi试验(失败,慎入)
2010-04-12T00:00:00+08:00
linux
http://chenlinux.com/2010/04/12/intro-iscsi-1
<p>聊天时听同事提及iscsi。回来后借助百度和谷歌大概了解了一下是网络存储,就赶紧下了软件包作测试。<br />
<code class="highlighter-rouge">bash
wget http://downloads.sourceforge.net/project/iscsitarget/iscsitarget/1.4.19/iscsitarget-1.4.19.tar.gz
tar zxvf iscsitarget-1.4.19.tar.gz
cd iscsitarget-1.4.19
make && make install
</code><br />
就完成了服务器端的安装,然后修改配置文件/etc/iet/ietd.conf,方便起见,就写最基本的三行:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Target iqn.2010-04.com.test:storage.xvdb1
Lun 0 Path=/dev/xvdb1,Type=fileio
Alias Test
</code></pre>
</div>
<p>启动服务/etc/init.d/iscsi-target start,netstat查看,发现有3260端口的监听了。</p>
<p>然后是客户端,更容易,直接yum install iscsi-initiator-utils就够了。然后启动服务/etc/init.d/iscsid start。</p>
<p>然后链接服务器,iscsiadm -m discovery -t st -p 10.12.13.86,可以搜索到了。不过居然出现两个,分别是服务器内外网ip搜索的值。两个办法。</p>
<p>一个是定义客户端/var/lib/iscsi/iface/ieth0文件如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>iface.iscsi_ifacename = ieth0
iface.net_ifacename = eth0
iface.hwaddress = default
iface.transport_name = tcp
</code></pre>
</div>
<p>然后用iscsiadm -m discovery -t st -p 10.12.13.86 -I ieth0命令搜索;</p>
<p>一个是定义服务端/etc/iet/targets.allow文件如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>ALL 10.12.13.0/24
</code></pre>
</div>
<p>然后/etc/init.d/iscsi-target restart就可以了。</p>
<p>最后搜索结果如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>10.12.13.86:3260,1 iqn.2010-04.com.rao:storage.xvdb1
</code></pre>
</div>
<p>在客户端用iscsiadm -m node -p 10.12.13.86 –targetname iqn.2010-04.com.rao:storage.xvdb1 –login命令,就可以连上磁盘了。然后用fdisk -l命令可以看到如下输出:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Disk /dev/sda: 750.1 GB, 750153729024 bytes
</code></pre>
</div>
<p>试验到目前为止一切正常。然后再mount这个sda的时候,赫然提示没有格式化!!<br />
好吧,fdisk /dev/sda然后mkfs.ext3 /dev/sda1再mount /dev/sda1 /mnt。<br />
OK,返回服务器端,mount,居然提示ext3错误~~</p>
<p>再开一台,同样到mount的时候,又一次提示没有格式化。<br />
难道iscsi和nfs不一样,压根不能多客户端同时连接一块磁盘???<br />
还是说因为iscsi默认是sda,而虚拟机是xvda,之间有冲突???<br />
于是返回谷歌搜索“iscsi xen”,结果看来应该是第一种了。<br />
<a href="http://www.performancemagic.com/iscsi-xen-howto/index.html">xen和iscsi的常见合用</a><br />
,是在Domain0上链接iscsi,然后基于不同的target分别create虚拟机模板和虚拟机磁盘,进一步可以保证虚拟平台出问题时,可以迅速的转移到另一个虚拟平台上。<br />
而后看到<a target="_blank" href="http://www.sansky.net/">存储部落</a>中的介绍,iscsi可以同样理解为裸设备,要达到网络共享,需要配套另外的共享软件~比如GFS</p>
客户页面小故障
2010-04-10T00:00:00+08:00
CDN
html
http://chenlinux.com/2010/04/10/refresh-problem-of-js-location_replace
<p>今天接到客户电话,说,主页已经提交过了刷新任务,上面的图片已经更新,但点击图片后链接到的页面内容还是老的……<br />
仔细一看,原来主页里的html是这么定义的:<br />
<code class="highlighter-rouge">html
<a href="http://msn.golfbox.cn/cache/922/36717.html" target="_blank"><img src="/upload/img/20100410/20100410144202.jpg" width="125" height="80" class="border_blue" /></a>
</code><br />
而http://msn.golfbox.cn/cache/922/36717.html的内容是这样:<br />
```html</p>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<script language="JavaScript">
location.replace("http://msn.golfbox.cn/cache/874/36804.html");
</script>
<div class="highlighter-rouge"><pre class="highlight"><code>于是跟客户交流确认,他们网站的主页框架上比较固定的地方,都是采用这种方式,来提供内容更新的。。。
他们又不肯给出具体更新了那些跳转页面,只好写个小脚本,去处理比对。如果有相同url的location.replace地址不统一的,就强制更新这些页面。
```bash
#!/bin/bash
curl http://msn.golfbox.cn/ |sed 's/\n//g'|sed 's/href=/\n/g'|grep "^"http://msn.golfbox.cn/cache"|awk -F'"' '{print $2}'>url
for j in `cat url`;do
for i in `cat ip`;do
curl -x $i:80 $j|awk -F'"' '/replace/{print "'$j'","'$i'",$2}'>>golfbox.log
# curl -x $i:80 -I $j|awk '/Age/{print "'$j'","'$i'",$2}'>>age.log
done
done
cat golfbox.log |awk '{if($1==a){if($3!=b){system("/home/squid/bin/squidclient -p 80 -h "$2" -m purge "$1)}};a=$1;b=$3}'
</code></pre>
</div>
<p>(该脚本有一个问题:如果刚好是第一台的url未更新,结果会反而去刷新那些已经更新了的服务器;所以应该是输出url,然后再用for do done去刷新所有服务器~如果curl可以在获取内容的同时获取Age就好了~所以最后用一个讨巧的办法:在ip列表的最开始放上客户源站的ip,这样就能保证最新了)<br />
以上是第一个。<br />
第二个:<br />
客户首页上还有一个问题:其右下角有跟随悬浮窗口,本来应该可以关闭的。但一点关闭,就提示javascrpt:iclose()错误页面。而用浏览器另存为html到桌面以后,再打开保存下来的页面,点击关闭就正常关闭了!<br />
经过查找,其中调用的js文件是http://msn.golfbox.cn/js/show.js,内容中关于javascrpt:iclose()的内容如下:<br />
<code class="highlighter-rouge">javascript
var sogouTall ='<div style="z-index:1000;position: absolute;display:none;" id="sogoubox">'
+'<a href="javascript:iclose();" style="float:left; margin-left:190px">关闭</a>'
+'<div style="clear:both">'
+'<a href="<a href="http://new.msn.golfbox.cn/wd/">http://new.msn.golfbox.cn/wd/"><img</a> src="<a href="http://new.msn.golfbox.cn/wd/dbtt.jpg">http://new.msn.golfbox.cn/wd/dbtt.jpg</a>" style="width:220px;height:160px"/></a>'
+ '<input type=hidden name="sogouAccountId" value="202014">'
+ ''
function iclose()
{
document.getElementById('sogoubox').style.display='none';
}
</code><br />
以我浅薄的web知识,是没看出来什么问题~~客户也莫名其妙,最好撤销掉这个窗口了</p>
url_rewrite_concurrency
2010-04-08T00:00:00+08:00
squid
http://chenlinux.com/2010/04/08/url_rewrite_concurrency
<p>squid的重定向,我看网上一般都采用 <code class="highlighter-rouge">redirect_children</code> (即 <code class="highlighter-rouge">url_rewrite_children</code>)。估计是因为中文权威指南的原因吧。不过中文权威指南还是2.5版的时候出的。有些新东西没有。比如 <code class="highlighter-rouge">squid.conf.default</code> 中提供的另一种 <code class="highlighter-rouge">url_rewrite_concurrency</code>。</p>
<p>官方使用说明如右:<a href="http://wiki.squid-cache.org/Features/Redirectors#How_do_I_make_it_concurrent.3F">http://wiki.squid-cache.org/Features/Redirectors#How_do_I_make_it_concurrent.3F</a></p>
<p>简单的说,就是开启 <code class="highlighter-rouge">url_rewrite_concurrency</code> 后,squid传递给rewriter的流由四个域增加为五个——最前头多了一个ID。然后rewriter返回的,也就有两个域,ID和uri。</p>
<p>简单修改一下原来的脚本如下即可:<br />
<code class="highlighter-rouge">perl
#!/usr/bin/perl -wl
use strict;
$|=1;
while () {
my ($id,$url,$client,$ident,$method) = ( );
($id, $url, $client, $ident, $method) = split;
if ($url =~m#^(.*)(?.*)#i) {
my ($domain,$option) = ($1,$2);
print "$id $domain\n";
} else {
print "$id\n";
}
}
</code><br />
然后squid.conf里修改如下:<br />
<code class="highlighter-rouge">squid
acl rewriteurl url_regex -i ^http://drag.g1d.net/.*.mp40drag?
url_rewrite_access deny !rewriteurl
url_rewrite_program /home/squid/etc/redirect.pl
#url_rewrite_children 10
url_rewrite_concurrency 10
</code></p>
date的一个怪问题
2010-04-03T00:00:00+08:00
bash
http://chenlinux.com/2010/04/03/a-unfound-option-of-date-command
<p>今天看同事的一个备份脚本,在取昨天的日期时,采用了YESTERDAY_NAME=<code class="highlighter-rouge">date -I -d'-1 day' +"%Y-%m-%d"</code>的方法。</p>
<p>我对这个方法比较感兴趣,赶紧试试,结果赫然返回“date: multiple output formats specified”的报错来!</p>
<p>而单独采用date -I -d’-1 day’或者date -d’-1 day’ +”%Y-%m-%d”这两种方法,都能返回正确结果。</p>
<p>返回同事的机器上,运行原命令确实没有问题。于是开始查版本,我的机器bash是3.2.25,date是5.97;他的bash是3.0.0.9,date是5.2.1。</p>
<p>不过在date -help中,不论是我的,还是他的机器上,都没发现-I这个option!!</p>
<p>为了移植性,同事也更改脚本删除了-I。不过之前脚本中为什么有这个,为什么还就真能跑,真是怪事~~</p>
lvs的activeconn与netstat的conn
2010-04-02T00:00:00+08:00
linux
http://chenlinux.com/2010/04/02/diff-between-lvs-active_conn-and-netstat-conn
<p>曾经有过一组高并发请求的服务器,在lvs上看到单台 activeconn 约等于 220000;同时,在RS上执行 <code class="highlighter-rouge">netstat -s -t|grep "connections established"</code> 结果大概是 65000,而 <code class="highlighter-rouge">squidclient mgr:5min|grep client_http.requests</code> 结果却只有 180。后来说起并发数的时候,有些茫然,到底哪个才算是呢?</p>
<p>今天在squid-user里问起,居然有幸碰见另一位中国订阅者,夏兄随后提供给我一个06年的帖子,所述甚详。算是解惑了~<br />
<a href="http://www.linuxsir.org/bbs/showthread.php?t=248500">http://www.linuxsir.org/bbs/showthread.php?t=248500</a></p>
<p>原来 lvs 默认有个超时时间,可以用 <code class="highlighter-rouge">ipvsadm -L --timeout</code> 查看,默认是 <code class="highlighter-rouge">900 120 300</code> ,分别是 <code class="highlighter-rouge">TCP TCPFIN UDP</code> 的时间。</p>
<p>而RS上参数如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>net.ipv4.tcp_keepalive_time <span class="o">=</span> 600
net.ipv4.route.gc_timeout <span class="o">=</span> 100
net.ipv4.tcp_fin_timeout <span class="o">=</span> 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close <span class="o">=</span> 10
net.ipv4.netfilter.ip_conntrack_tcp_timeout_max_retrans <span class="o">=</span> 300
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait <span class="o">=</span> 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack <span class="o">=</span> 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait <span class="o">=</span> 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait <span class="o">=</span> 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established <span class="o">=</span> 300
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv <span class="o">=</span> 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent <span class="o">=</span> 120
net.ipv4.netfilter.ip_conntrack_udp_timeout_stream <span class="o">=</span> 180
net.ipv4.netfilter.ip_conntrack_udp_timeout <span class="o">=</span> 30
</code></pre>
</div>
<p>可见 RS 的超时时间(仅指 <code class="highlighter-rouge">ESTABLISHED</code> )比 LVS 小了 3 倍。再加上 <code class="highlighter-rouge">WAIT</code> 和 <code class="highlighter-rouge">SYN_RECV</code> 等状态,差不多就是 220000/65000 的比例了。而 squid 的 RPS 是按秒计算的,<code class="highlighter-rouge">180*300=~55000</code> ,在数量级上和 netstat 的结果也就差不多了~</p>
perl边学边练(purge脚本)
2010-03-30T00:00:00+08:00
CDN
perl
squid
http://chenlinux.com/2010/03/30/purge-script-by-perl
<p>squid的purge,一般有两种方式,squidclient -m purge url或者http request (method)purge url。如果任务不太多的情况下,直接使用squidclient -p 80 -h 1.2.3.4 -m purge url即可。如果任务比较繁重的情况下,telnet80后直接发送purge请求稍微好一些。作为初学perl的练手,写一个purge脚本。如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/perl -w</span>
<span class="k">use</span> <span class="nn">IO::</span><span class="nv">Socket</span><span class="p">;</span>
<span class="c1">#检测脚本参数个数</span>
<span class="k">unless</span> <span class="p">(</span><span class="nv">@ARGV</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span> <span class="nb">die</span> <span class="s">"usage: $0 url"</span> <span class="p">}</span>
<span class="c1">#打开ip列表文件,定义文件句柄HOST</span>
<span class="nb">open</span><span class="p">(</span><span class="nv">HOST</span><span class="p">,</span><span class="s">"./ip"</span><span class="p">);</span>
<span class="c1">#定义连接结束符,然后翻倍(汗这个方式~)</span>
<span class="nv">$EOL</span> <span class="o">=</span> <span class="s">"1512"</span><span class="p">;</span>
<span class="nv">$BLANK</span> <span class="o">=</span> <span class="nv">$EOL</span> <span class="nv">x</span> <span class="mi">2</span><span class="p">;</span>
<span class="c1">#从打开的文件中读取具体ip</span>
<span class="c1">#@host = HOST;</span>
<span class="c1">#my $ip;</span>
<span class="c1">#foreach $ip(@host){</span>
<span class="c1">#当ip列表较大时,采用@host的方法可能out of memory,所以采取逐行读取</span>
<span class="k">while</span> <span class="p">(</span><span class="nb">defined</span><span class="p">(</span><span class="nv">$ip</span><span class="o">=</span><span class="sr"><HOST></span><span class="p">)){</span>
<span class="c1">#从参数中默认读取url变量</span>
<span class="k">foreach</span> <span class="nv">$document</span> <span class="p">(</span> <span class="nv">@ARGV</span> <span class="p">)</span> <span class="p">{</span>
<span class="c1">#利用IO::Socket::INET模块定义TCP80连接,port可以读取/etc/services文件里的定义</span>
<span class="nv">$remote</span> <span class="o">=</span> <span class="nn">IO::Socket::</span><span class="nv">INET</span><span class="o">-></span><span class="k">new</span><span class="p">(</span> <span class="nv">Proto</span> <span class="o">=></span> <span class="s">"tcp"</span><span class="p">,</span>
<span class="nv">PeerAddr</span><span class="err"> </span> <span class="o">=></span> <span class="nv">$ip</span><span class="p">,</span>
<span class="nv">PeerPort</span><span class="err"> </span> <span class="o">=></span> <span class="s">"http(80)"</span><span class="p">,</span>
<span class="p">);</span>
<span class="k">unless</span> <span class="p">(</span><span class="nv">$remote</span><span class="p">)</span> <span class="p">{</span> <span class="nb">die</span> <span class="s">"cannot connect to http daemon on $ip"</span> <span class="p">}</span>
<span class="c1">#立即输出</span>
<span class="nv">$remote</span><span class="o">-></span><span class="nv">autoflush</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="c1">#向定义的TCP连接文件句柄发送purge请求</span>
<span class="c1">#这里可以直接"\n\n",不过采用"".$BLANK的方式可能规范一些,因为在win或者mac的平台上,是不一样的</span>
<span class="k">print</span> <span class="nv">$remote</span> <span class="s">"PURGE $document HTTP/1.0"</span> <span class="o">.</span> <span class="nv">$BLANK</span><span class="p">;</span>
<span class="k">while</span> <span class="p">(</span> <span class="sr"><$remote></span> <span class="p">)</span> <span class="p">{</span> <span class="k">print</span> <span class="p">}</span>
<span class="c1">#关闭tcp连接</span>
<span class="nb">close</span> <span class="nv">$remote</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1">#关闭ip列表文件</span>
<span class="nb">close</span><span class="p">(</span><span class="nv">HOST</span><span class="p">);</span>
</code></pre>
</div>
squid源站故障转向(终结篇)
2010-03-28T00:00:00+08:00
CDN
squid
C
http://chenlinux.com/2010/03/28/modify-error_page-src-for-location
<p>因为这么一个想法,我陆陆续续的把squid很多功能都理了一遍,今天终于打算写个不完美的终结篇。而就在写这个终结篇的同时,公司里也已经开始把这批别扭的客户改往nginx平台加速了。</p>
<p>总结这批客户的跳转要求,其实格式都比较统一,大抵就是*.abc.com(.cn)坏了就转到abc.cdn.21vokglb.cn。在3月24日的博文最后,已经有了一个思路——既然无法执行php的header(Location)和strstr(%U),那么就干脆在squid的src里对%U进行操作好了。</p>
<p>squid-src/errorpage.c中关于%U的注释是:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>U - URL without password
</code></pre>
</div>
<p>相关语句是:<br />
<code class="highlighter-rouge">c
p = r ? urlCanonicalClean(r) : err->url ? err->url : "[no URL]";
</code><br />
只要把url按”.”分割,然后取出第二个域abc,就可以在html代码中给它加上跳转后的url了——这一步也能在src里完成,不过以后不好修改了,虽然现在这样子的定制性也强不到哪去~<br />
从大二到现在无数年了,c已经属于忘到冥王星外的东东,于是一个一个的翻c的字符串函数,从strstr、strchr、strcat、strtok、strsep到最后终于发现sscanf。只要在src/errorpage.c的588行下加这么一句话就可以了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="mi">587</span><span class="err"> </span> <span class="k">case</span> <span class="sc">'U'</span><span class="p">:</span>
<span class="mi">588</span><span class="err"> </span> <span class="n">p</span> <span class="o">=</span> <span class="n">r</span> <span class="o">?</span> <span class="n">urlCanonicalClean</span><span class="p">(</span><span class="n">r</span><span class="p">)</span> <span class="o">:</span> <span class="n">err</span><span class="o">-></span><span class="n">url</span> <span class="o">?</span> <span class="n">err</span><span class="o">-></span><span class="n">url</span> <span class="o">:</span> <span class="s">"[no URL]"</span><span class="p">;</span>
<span class="mi">589</span><span class="o">+</span><span class="err"> </span> <span class="n">sscanf</span><span class="p">(</span><span class="n">p</span><span class="p">,</span><span class="s">"%*[^.].%[^.]"</span><span class="p">,</span><span class="n">p</span><span class="p">);</span>
<span class="mi">590</span><span class="err"> </span> <span class="k">break</span><span class="p">;</span>
</code></pre>
</div>
<p>然后编译安装,一路通过没有问题~~启动squid服务,测试一下%U吧~<br />
先把ERR_ACCESS_DENIED内容修改如下:<br />
```html</p>
<html>
<body>
<head>
<META HTTP-EQUIV="refresh" Content="0;URL=http://%U.cdn.21vokglb.cn/index.htm">
</head>
```
然后在squid.conf中增加对自己本机的访问控制如下:
```squid
acl test src 222.62.104.189/255.255.255.255
http_access deny rao
```
squid -k reconfigure生效,访问一下www.xyfunds.com.cn,果然跳转到xyfunds.cdn.21vokglb.cn/index.htm啦~~
完毕。
虽然最终还是没能达到任意定义跳转url的目标,不过就本身的出发点来说,还是完成了需求。这也是我N年来第一次重新看C,也是第一次修改squid代码,虽然只加了一句~~~不过意义还是有滴,晚上吃个鸡蛋自我犒劳一下咯。
</body></html>
zabbix_proxy部署
2010-03-25T00:00:00+08:00
monitor
http://chenlinux.com/2010/03/25/zabbix_proxy-install
<p><em>continue</em></p>
<p>zabbix作为分布式监控系统,不试试实在可惜。好在做起来也简单。 <br />
首先要求编译时有enable-proxy参数,这个已经有了; <br />
然后修改zabbix_proxy.conf,和zabbix_server.conf相同的修改(DB等)就不再说了: <br />
Server=要写最上层zabbix的ip <br />
Hostname=要写独一无二的,在最上层zabbix的web配置上要用 <br />
ConfigFrequency=这个是配置同步时间差,设短一点,默认3600太长 <br />
TrapperTimeout=超时时间,设短一点,默认300,最好不超过30</p>
<p>然后在最上层的zabbix的web界面上添加proxy即可。administrator-DM-proxy-Add,填入刚才独一无二的那个Hostname即可。</p>
<p>Add Host的时候,选择proxy——host里agentd.conf的Server也必须是相应proxy的ip才行。 <br />
<em>To_be_continued</em></p>
cacti优化
2010-03-25T00:00:00+08:00
monitor
cacti
http://chenlinux.com/2010/03/25/cacti-optimization
<p>首先,采用spine代替cmd.php来采集数据。<br />
下载与cacti相应版本的spine和补丁:<br />
<code class="highlighter-rouge">bash
wget http://www.cacti.net/downloads/spine/cacti-spine-0.8.7e.tar.gz
tar zxvf cacti-spine-0.8.7e.tar.gz -C /tmp
cd /tmp/cacti-spine-0.8.7e
wget http://www.cacti.net/downloads/spine/patches/snmp_v3_fix.patch
wget http://www.cacti.net/downloads/spine/patches/mysql_client_reconnect.patch
wget http://www.cacti.net/downloads/spine/patches/ping_reliability.patch
patch -p1 -N < snmp_v3_fix.patch
patch -p1 -N < mysql_client_reconnect.patch
patch -p1 -N < ping_reliability.patch
./configure –prefix=/cache/data/cacti --with-mysql=/home/mysql
make
# 这个时候发现报错了,spine0.8.7e编译安装要求最新版本的automake1.11,于是去下automake:
wget http://ftp.gnu.org/gnu/automake/automake-1.11.tar.gz
tar zxvf automake-1.11.tar.gz -C /tmp/
cd /tmp/automake-1.11/
./configure --prefix=/usr
# 又报错,automake1.11要求新版本的autoconf2.64,于是去下autoconf:
wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.65.tar.gz
tar zxvf autoconf-2.65.tar.gz -C /tmp/
cd /tmp/autoconf-2.65/
./configure --prefix=/usr && make && make install
</code></p>
<p>返回安装automake,即可成功;安装spine,也成功了。 <br />
修改/cache/data/cacti/etc/spine.conf里的db信息,和cacti的global.php里一致。 <br />
登陆web界面,settings中修改poller type为spine,修改Spine Specific Execution Parameters里的Maximum Threads per Process为cpu数的2倍。save~ <br />
第二、给cacti-tables建index。默认的cacti.sql里一个index索引都没有~ <br />
<code class="highlighter-rouge">sql
CREATE INDEX `data_template_data_id` ON `data_input_data` (`data_template_data_id`);
CREATE INDEX `host_id_snmp_query_id_snmp_index` ON data_local (`host_id`,`snmp_query_id`,`snmp_index`);
CREATE INDEX `local_data_id_data_source_name` ON data_template_rrd (`local_data_id`,`data_source_name`);
CREATE INDEX `graph_template_id_local_graph_id` ON graph_templates_item (`graph_template_id`,`local_graph_id`);
CREATE INDEX `local_graph_template_item_id` ON graph_templates_item (`local_graph_template_item_id`);
CREATE INDEX `host_id_snmp_query_id_snmp_index` ON host_snmp_cache (`host_id`,`snmp_query_id`,`snmp_index`);
CREATE INDEX `local_data_id_rrd_path` ON poller_item (`local_data_id`,`rrd_path`);
CREATE INDEX `host_id_rrd_next_step` ON poller_item (`host_id`,`rrd_next_step`);
CREATE INDEX host_id_snmp_query_id ON host_snmp_cache (host_id,snmp_query_id);
CREATE INDEX host_id_snmp_port ON poller_item (host_id,snmp_port);
CREATE INDEX data_source_path ON data_template_data (data_source_path);
</code></p>
<p>第三、重构rra目录结构。按照device分结构。 <br />
/home/php/bin/php /cache/data/cacti/cli/structure_rra_paths.php –proceed即可。 <br />
web界面中settings里的Paths可以勾选Structured RRA Path(/host_id/local_data_id.rrd)即可。</p>
<p>最后,据这个优化的<a target="_blank" href="http://zys.8800.org/index.php/archives/391/comment-page-1#comment-22">原作者</a>说,按此步骤,710台服务器,24000个RRD文件,完成一次poller.php的时间,缩短到50 seconds。</p>
cacti自建tcp80连接数监控
2010-03-24T00:00:00+08:00
monitor
cacti
http://chenlinux.com/2010/03/24/monitor-tcp-conns-by-cacti
<p>同样作为提供web服务的机器,因为不同业务的关系,除了流量以外,还需要参考TCP80连接数来分析服务器性能状况。下面就试试cacti对连接数的监控。<br />
最简单的方法,利用snmpnetstat这个命令,自动搞定一切。cactiuser.org上提供一个现成的模板,只要import就能直接用。下载地址如右:<a href="http://www.iammecn.com/wp-content/uploads/2009/12/cacti_graph_template_snmp_connections.zip" target="_blank">http://www.iammecn.com/wp-content/uploads/2009/12/cacti_graph_template_snmp_connections.zip</a></p>
<p>不过这个snmpnetstat用起来精准度不太好保证,比如我在测试机上运行如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[root@BeiJingBGP-Dns-02 ~]# snmpnetstat -c 'public' -v 2c 10.10.10.43:161
Active Internet (udp) Connections
Proto Local Address
udp *.ntp
udp *.snmp
udp *.filenet-
udp *.54927
udp localhost.domain
udp localhost.ntp
udp localhost.domain
udp localhost.ntp
udp 211.151.65.53.ntp
udp 211.151.67.80.domain
</code></pre>
</div>
<p>居然一个tcp都没显示出来。我汗~~<br />
所以采用一个自建脚本模板的方法来完成吧~~<br />
首先在host上新建/etc/snmp/tcpconn.sh,内容如下:<br />
<code class="highlighter-rouge">bash
#!/bin/sh
conn=`netstat -s -t | grep connections established |awk '{print $1}'`
echo $conn
</code></p>
<table>
<tbody>
<tr>
<td>对这个脚本我个人持保留意见。因为在netstat -anpl</td>
<td>grep :80</td>
<td>wc -l、netstat -s -t和/proc/net/tcp中来看,netstat -s -t最省时间,但数字也最不准~</td>
</tr>
</tbody>
</table>
<p>三种方法测试如下:<br />
[root@BeiJingBGP-Dns-02 ~]# time netstat -s -t|awk ‘/connections established/{print $1}’<br />
3</p>
<div class="highlighter-rouge"><pre class="highlight"><code>real 0m0.008s
user 0m0.000s
sys 0m0.008s
[root@BeiJingBGP-Dns-02 ~]# time netstat -plna|awk '/:80/{a++}END{print a}'
1
real 0m0.065s
user 0m0.000s
sys 0m0.048s
[root@BeiJingBGP-Dns-02 ~]# time awk '$4=="01"{a++}END{print a}' /proc/net/tcp
real 0m0.023s
user 0m0.000s
sys 0m0.024s
</code></pre>
</div>
<table>
<tbody>
<tr>
<td>(话说顺便试了一下</td>
<td>grep :80</td>
<td>wc -l和</td>
<td>awk ‘/:80/{a++}END{print a}’的time,awk快5ms,哈哈~)</td>
</tr>
</tbody>
</table>
<p>然后在/etc/snmp/snmpd.cong里添加如下句:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>exec .1.3.6.1.4.1.2021.18 tcpCurrEstab /etc/snmp/tcpconn.sh
</code></pre>
</div>
<p>重启snmpd服务即可。</p>
<p>回cacti服务器端,运行如下命令,可以看到相关输出即可。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[root@BeiJingBGP-Dns-02 ~]# snmpwalk -c 'public' -v 2c 10.10.10.43 .1.3.6.1.4.1.2021.18
UCD-SNMP-MIB::ucdavis.18.1.1 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.18.2.1 = STRING: "tcpCurrEstab"
UCD-SNMP-MIB::ucdavis.18.3.1 = STRING: "/etc/snmp/tcpconn.sh"
UCD-SNMP-MIB::ucdavis.18.100.1 = INTEGER: 0
UCD-SNMP-MIB::ucdavis.18.101.1 = STRING: "3"
UCD-SNMP-MIB::ucdavis.18.102.1 = INTEGER: 0
UCD-SNMP-MIB::ucdavis.18.103.1 = ""
</code></pre>
</div>
<p>这个string:”3”就是真正的tcp80连接数。<br />
然后进入cacti的web页面进行设置吧:</p>
<table>
<tbody>
<tr>
<td>在cacti界面中console->Templates->Data Templates,然后点击右上角的Add,Data Templates中的name是给这个数据模板的命名,Data Source中的name将来显示在Data Sources中,我这里添加“</td>
<td>host_description</td>
<td>– Tcp Conn. – ESTBLISHED”,选get snmp data,Internal Data Source Name也可以随便添,这个用来给rrd文件命名。设置完后就可以create了,之后会发现下面多了一些选项,在最下面那个添上我们需要的数据的OID“.1.3.6.1.4.1.2021.18.101.1”,可以save了。</td>
</tr>
</tbody>
</table>
<table>
<tbody>
<tr>
<td>此后需要创建一个Graph Templates,好让cacti生成图片。在cacti界面中console->Templates->Graph Templates,然后点击右上角的Add,Templates中的name是给这个数据模板的命名,Graph Template中的name是将来显示在图片上面中间的内容,我这里添加“</td>
<td>host_description</td>
<td>– Tcp Conn. –ESTBLISHED”,其他保持默认,保存之后上面会出来一些选项。在Graph Template Items中添加一个item,Data Source选之前添加的,color选择一个图片的颜色,Graph Item Type选AREA,也就是区域,也可以选其他的线条,Text Format设置说明。然后再添加一个,Graph Item Type选GPRINT,Consolidation Function选LAST,也就是当前的值,Text Format输入current。你还可以添加一些Graph Item Type为COMMENT的注释说明等。</td>
</tr>
</tbody>
</table>
<p>最后,可以把graph加进host templates,或者直接在device中加graph。过一会就能看到图啦~~~</p>
squid和nginx的error_page差别
2010-03-24T00:00:00+08:00
CDN
squid
nginx
http://chenlinux.com/2010/03/24/diff-of-error_page-between-squid-nginx
<p>nginx的error_page,有两个种办法。</p>
<p>一是直接error_page 502 503 504 = http://xyfunds.cdn.21vokglb.cn/index.htm;</p>
<p>二是error_page 502 503 504 =200 @fetch;然后location ~* @fetch {……}。</p>
<p>网上看到很多自定义error_page的方式,比如error_page 404 /404.php;把变量都传给php去分析处理;也可以在location里用if(){}做rewrite等等。</p>
<p>squid的error_page,可以有error_directory、error_map、deny_info三种方式。其中deny_info仅适用于ERR_ACCESS_DENIED一种情况;目前对于源站故障跳转,采用的是修改error_directory里html的meta;今天由nginx的方式想到采用error_map试试,于是写了如下php页面:<br />
<code class="highlighter-rouge">php
<?php
switch ($_SERVER['SERVER_NAME'])
{
case 'www.xyfunds.com.cn':
header("Location: http://xyfunds.cdn.21vokglb.cn/index.htm");
break;
default:
echo $_SERVER['SERVER_NAME'];
}
?>
</code><br />
但测试结果,ERR的页面却是一片空白……在squid.conf,default中看到,原来<span style="color:#ff0000;">error_map只是返回定义页面的内容,header还是原先的。</span><br />
于是又想针对squid返回的%U,进行strtr(),然后再进行meta,如下:<br />
<code class="highlighter-rouge">php
<?php
$urlarray = array('com.cn'=>'21vokglb.cn');
$urlrewrite = strtr("%U",$urlarray);
echo <META HTTP-EQUIV="refresh" CONTENT="0; $urlrewrite">;
?>
</code><br />
很可惜,测试结果是把这串字符直接显示在了页面上。<br />
这两个结果让我很无语。如果说squid不支持php,那为什么它能识别出来上一个是修改header而不是页面内容所以不显示文本;如果说squid支持php,那为什么下一个又无法执行呢?</p>
<p>想对%U进行操作,难道非得到squid/src/errorpage.c里去修改么?C语言的字符串处理没有封装好的函数,真的好麻烦的说……</p>
nginx的默认主机头问题
2010-03-24T00:00:00+08:00
nginx
http://chenlinux.com/2010/03/24/default-vhost-in-nginx
<p>今天发现nginx做多域名混跑的proxy_cache时有一个小问题:当一个非加速server_name的请求到达的时候,nginx不会像squid那样返回一个ERR_DNS_FAIL,反而假装很正常的返回一个页面:<br />
<code class="highlighter-rouge">nginx
http {
server {
server_name www.aaa.com;
proxy_pass http://1.1.1.1;
}
server {
server_name www.bbb.com;
proxy_pass http://2.2.2.2;
}
}
</code><br />
访问www.ccc.com的时候,nginx毫不犹豫的把www.aaa.com的页面内容返回给了client。在日志里记录:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>MISS/200 GET http://www.ccc.com/ 1.1.1.1:80
</code></pre>
</div>
<p>如果把aaa和bbb的server{}顺序倒换,那ccc的回源地址就变成了2.2.2.2……</p>
<p>也就是说,解析不出、定义不到的域名请求,自动返回排在第一位的server内容。</p>
<p>解决办法也容易,在最上头,也定义一个自己的server,就可以了:<br />
<code class="highlighter-rouge">nginx
http {
server {
root /cache;
rewrite ^(.*) /error.htm permanent;
}
server {
……
}
}
</code><br />
至于这个error.htm,有没有都一样,反正有就是301,没有就是404~~</p>
<p>后面的web服务器为什么不检查Host直接给出内容,也是让人很郁闷的一点。或许IIS就是如此?</p>
fms3.5试用
2010-03-21T00:00:00+08:00
CDN
http://chenlinux.com/2010/03/21/fms3-5
<p>flash和flex应该是现在网络上红到发紫的技术了,要flv不要tv,要sns不要bbs,连春晚节目都满嘴偷菜~~~我也试着用用fms3.5,一些基本搭建,过过瘾,免得自己太过OUT了。 <br />
fms是adobe的付费产品,不过不花钱买序列号,也可以用免费的开发版,限制是10个链接而已。上www.adobe.com注册一个帐号,就可以下载了。下载下来的zip中包含了win版的exe和linux版的tar.gz。网上教程大多是win的,不过不要紧,除了字符和图形,真正使用上没啥区别。 <br />
上传FlashMediaServer3.5.tar.gz到linux服务器上,解压开,./installFMS -platformWarnOnly,然后按提示,设定安装目录、管理员帐号密码、监听端口、是否安装自带的apache2.2.9,是否设为守护进程等,最后完成。 <br />
ps看看,已经有几个fms的进程了吧;netstat看看,也有1935/19350/1111端口的监听吧(前提是你没改),OK了。 <br />
注:这里有几个问题。如果install的时候,端口选择了默认的1935,80,而本身服务器上又开着httpd的80,就会有冲突;这时候关掉FMSHttpd是不够的,因为fmsedge也监听这80端口,需要在conf/fms.ini中,把SERVER.HTTPD_ENABLED设为false,再删除ADAPTOR.HOSTPORT里的80,然后/etc/init.d/fms restart,这样就可以了。</p>
<ul>
<li>VOD点播:</li>
</ul>
<p>上传一个flv/mp4/avi/f4v到服务器上,放进applications/vod/media/目录里。然后用播放器播放rtmp://testip/vod/sample就可以播放flv格式的视频了,如果是mp4/avi/f4v等H.264的视频,uri地址为rtmp://testip/vod/mp4:sample.f4v,也就是申明一下它是mp4的,然后写全文件名就行。 <br />
fms自带一个测试的页面,可以用来试验。在apache里建一个virtualhost,发布/opt/adobe/fms/samples/videoPlayer,就可以用http://testip/videoPlayer/videoplayer.html访问这个测试页面,把上面说到的url贴进STREAM URL:里,选择VOD,点击PLAY STREAM,看到上面出现loading、buffering,然后就开始播放上传的那个视频了。OK~ <br />
rtmp流方式和nginx/lighttpd的flv_module不同的是,rtmp的拖动,你拖到哪它就从哪开始播放;而flv_module的拖动,只是在你拖到的地点附件找一个最接近的关键帧,向服务器发送start=***的query_strings,然后从该关键帧开始播放。这也就意味着在拖动之前,你至少得等到flv的meta信息(主要是全部的关键帧位置)下载完才行。</p>
<ul>
<li>LIVE直播:</li>
</ul>
<p>上面的小实验完成了,可能惊喜不是很大,唉,谁让web发布flv的拖动功能一样ok呢。嗯,下面开始用live功能,把web给ko掉~~ <br />
要完成直播,还得用另外一样东东:FME,全称flash media live encoder,同样上adobe官网,登陆帐号下载即可。两个消息,一好一坏,好消息是这个东东免费,不收钱;坏消息是这个东东只有windows版,口年那些用ubuntu、mac、fc的童鞋们了~~ <br />
下载安装运行,可以看到一个很一目了然的界面,IO画面,视频编码和分辨率选择、音频码率大小选择、保存地址、流服务器地址,日志。主要就是这些。 <br />
先试试最基本的保存视频,取消stream to flash media server的勾选,点击start,稍后再点stop,去我的文档里看我的视频文件夹,里面是不是就有个sample.flv了?那就成功了。(不行?看log排查,不会是你压根没接摄像头吧?) <br />
然后就是给fms服务器传直播流了。把刚才取消的勾重新打上。修改FMS url地址为服务器的正确地址,也就是把localhost改成testip,然后修改stream为一个别的名字(比如otherlivestream),因为livestream这个名字服务器上默认已经存在了。点connect,嗯,连不上…… <br />
这是因为fme和fms之间还差了个桥梁。返回刚才下载fme的页面(没关吧~),看到下面还有个auth-addin下载吧,赶紧下下来传服务器上。解压出来,./installSAA,同样是字符界面,一顿y就可以了,不过如果你前面修改过安装路径什么的,也得一样修改这里~ <br />
然后进入conf路径,里面除了fms.ini、Loggerxml、Server.xml、Users.xml外,多了users、users.dat两个文件。./users add -u username -p password,添加用户即可。然后你就可以看到users.dat中多了一行username:<em>**</em> <br />
返回本机的fme,再重点connect,很快弹出验证框,输入username和password,连接成功了,点start,直播流就此建立。 <br />
继续使用刚才看vod的页面,在stream url里填rtmp:/live/otherlivestream,选择live,点play stream,缓冲完成后,看是不是就是摄像头前的自己在动在说话?OK啦~</p>
zabbix安装试用
2010-03-18T00:00:00+08:00
monitor
http://chenlinux.com/2010/03/18/zabbix-install
<p>在CU上看到有帖子比较各开源monitor软件,其中对zabbix颇多赞誉。决定试用一下。<br />
<code class="highlighter-rouge">bash
#为快速安装方便,LAMP环境都采用yum获取。
yum install httpd mysql* php* gcc net-snmp* curl*
#然后编译安装zabbix,步骤如下:
groupadd zabbix
useradd -g zabbix zabbix
wget http://cdnetworks-kr-1.dl.sourceforge.net/project/zabbix/ZABBIX%20Latest%20Stable/1.8.1/zabbix-1.8.1.tar.gz
tar zxvf zabbix-1.8.1.tar.gz
cd zabbix-1.8.1
mysql -uroot -p
>create database zabbix;
>grant all privileges on zabbix.* to zabbix@"localhost" identified by '123456';
>flush privileges;
mysql -uroot -p zabbix < create/schema/mysql.sql
mysql -uroot -p zabbix < create/data/data.sql
mysql -uroot -p zabbix < create/data/images_msql.sql
./configure --prefix=/home/zabbix --enable-server --enable-proxy --enable-agent --with-mysql --with-net-snmp --with-libcurl
make && make install
cat >> /etc/services <<EOF
zabbix-agent 10050/tcp # Zabbix Agent
zabbix-agent 10050/udp # Zabbix Agent
zabbix-trapper 10051/tcp # Zabbix Trapper
zabbix-trapper 10051/udp # Zabbix Trapper
EOF
make /home/zabbix/conf
cp misc/conf/* /home/zabbix/conf/
ln -s /home/zabbix/conf /etc/zabbix
chown -R zabbix.zabbix /etc/zabbix
i=`hostname`;sed -i "s/^Hostname=system.uname/Hostname=$i" /etc/zabbix/zabbix_agentd.conf
sed -i 's/^DBUser=root/DBUser=zabbix/g' /etc/zabbix/zabbix_server.conf
sed -i 's/^# DBPassword=/DBPassword=zabbix/g' /etc/zabbix/zabbix_server.conf
mv frontends/php /var/www/html/zabbix
chown -R zabbix.zabbix /var/www/html/zabbix
</code><br />
根据zabbix需求修改LAMP,vi /etc/php.ini修改相关参数如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>max_execution_time = 300
date.timezone = Asia/Shanghai
post_max_size = 32M
memory_limit = 128M
mbstring.func_overload = 2
</code></pre>
</div>
<p>vi /etc/httpd/conf/httpd.conf修改servername,然后启动apache。</p>
<p>服务器操作部分完成,接下来是web配置。</p>
<p>浏览器打开http://mydomain.com/zabbix,出现setup.php,按说明next即可。到最后要求下载zabbix.conf.php。其实可以直接修改服务器文件。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>sed -i <span class="s1">'s/0";/3306";/g'</span> /var/www/html/zabbix/conf/zabbix.conf.php.example
sed -i <span class="s1">'s/_password//g'</span> /var/www/html/zabbix/conf/zabbix.conf.php.example
mv /var/www/html/zabbix/conf/zabbix.conf.php.example /var/www/html/zabbix/conf/zabbix.conf.php
</code></pre>
</div>
<p>然后test即可OK,进行登陆界面。初始用户名密码为admin/zabbix。<br />
进去以后,第一件事改密码。<br />
选择administrator下的user,点击admin,change password即可;还可以添上email等信息;另外,可以选择chinese,save之后relogin,可以看到稍微友好一点点的中文界面,不过翻译水平就请将就一下吧~~(最无语的是把select翻译成搜索==!)<br />
<a href="http://www.hiadmin.com" target="_blank">架构研究室</a>刚刚发布了一个汉化全面一些的<a href="http://www.hiadmin.com/wp-content/uploads/2010/03/cn_zh.inc.php_.tar.gz" target="_blank">语言包</a>,可以解压覆盖/var/www/html/zabbix/include/locales/cn_zh.inc.php。<br />
开始在页面上点点看看吧,不过这时候才想起来,web虽然开了,zabbix服务本身却一直没有启动呢~返回服务器操作:<br />
<code class="highlighter-rouge">bash
cp misc/init.d/redhat/zabbix_* /etc/init.d/
</code><br />
vi /etc/init.d/zabbix_server_ctl,把BASEDIR改成/home/zabbix,PIDFILE=/var/tmp修改成PIDFILE=/tmp/,$BASEDIR/bin/改成$BASEDIR/sbin/。zabbix_agentd_ctl同理。</p>
<p>然后,启动服务,/etc/init.d/zabbix_server_ctl start;/etc/init.d/zabbix_agentd_ctl start</p>
<p>(网上都没写pid路径也要改,实际上zabbix_server.conf里的路径是/tmp/zabbix_server.pid,不改会导致启动脚本失效)</p>
<p>zabbix和nagios等监控一样,需要在被监控host上安装agent来完成数据采集和其他操作。当然,如果就是不肯安装的话,也可以使用snmp来完成一些基本的东西。<br />
host上的agent部署特别简单:<br />
<code class="highlighter-rouge">bash
wget http://www.zabbix.com/downloads/1.8/zabbix_agents_1.8.linux2_6.x64.tar.gz
tar zxvf zabbix_agents_1.8.linux2_6.x64.tar.gz -C /home/
cat >> /home/zabbix/conf/zabbix_agentd.conf << eof
LogFile=/tmp/zabbix_agentd.log
Server=zabbix服务器的ip
Hostname=被监控host的名字
eof
</code><br />
然后启动即可。/home/zabbix/sbin/zabbix_agentd -c /home/zabbix/conf/zabbix_agentd.conf &</p>
<p>据称这里如果不写全路径,可能出错。</p>
<p>进入使用环节咯。大体上monitor都是这样,groups-hosts-templates-items-triggers-graphs-actions<br />
create一个新host group,太简单,略过; <br />
create一个新host,主要选择group、link相应的template; <br />
返回host列表,就可以选择它们自己的items、trigger、graph了。需要注意的是原有的trigger是从template里读出来的,所以修改的话也要从template里修改。 <br />
action是报警方式,比如上面编辑user时加的email,这里可以在add operations的时候选择send message给某user(不要忘了在administration的media types里配smtp),还可以选择remote command;另外add conditions的时候,选择具体是哪些主机、那些trigger等等。 <br />
graph是绘图,有阴影、连线、散点等方式,只要是items里有的,都能绘图出来。重叠在一张图里显示的时候,不要忘了改成好区分的颜色,一目了然~~常见的自然是流量、负载、磁盘使用等等图。 <br />
web是对url的监控,这个比nagios等要强大多了。因为使用libcurl,它可以模拟各种user-agent下的访问,还能post数据模拟一系列操作(登陆、发帖、收藏等等,先在variables里定义好变量,然后在step中传递)。添加完成后,就可以看到对这些url的响应速度和时间的监控图了。完成后,在该host的trigger中,select items还增加了web monitor项,分别是速度、时间、状态码,也可以对此进行报警。 <br />
<em>to_be_continued</em></p>
squid 灵异日志
2010-03-17T00:00:00+08:00
squid
http://chenlinux.com/2010/03/17/problem-of-squid-access-log
<p>今天在 squid 服务器上,无意看到一个让我无比惊讶的访问日志,随后一统计,同样的日志居然还不在少数,……</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="o">[</span>root@tinysquid2 ~]# tail -f /cache/logs/access.log |grep HIT
1268824378.683 64 125.39.107.46 TCP_MEM_HIT/200 851 GET http://www.114.com.cn/style/css/jquery.autocomplete.css - DIRECT/117.25.130.146 text/css <span class="s2">"http://www.114.com.cn/gindex.html"</span> <span class="s2">"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QQPinyin 686; SV1; 360SE)"</span>
</code></pre>
</div>
<p>居然存在 <code class="highlighter-rouge">TCP_MEM_HIT/200</code> 的情况下还 <code class="highlighter-rouge">DIRECT</code> 回源的情况!!</p>
<p>有同事猜测可能是当源站数据取回来后,因为 <code class="highlighter-rouge">cache_mem</code> 不够大,要转写到 <code class="highlighter-rouge">cache_disk</code> 上去,但请求并发较大,数据还没转写呢,mem 已经不足了,于是把相对较老的抛弃掉。于是在索引中标记这部分数据是 HIT 的,但实际又没有数据在 MEM 中,所以继续 DIRECT 回源取了。</p>
<p>不过从 squidclient 的 mgr:info 来看,不支持他的这种说法。</p>
<div class="highlighter-rouge"><pre class="highlight"><code> Cache information <span class="k">for </span>squid:
Request Hit Ratios: 5min: 45.4%, 60min: 55.7%
Byte Hit Ratios: 5min: 59.2%, 60min: 62.3%
Request Memory Hit Ratios: 5min: 30.1%, 60min: 24.4%
Request Disk Hit Ratios: 5min: 26.8%, 60min: 24.0%
Storage Swap size: 354184 KB
Storage Mem size: 32196 KB
Mean Object Size: 15.91 KB
Requests given to unlinkd: 498
</code></pre>
</div>
<p>可以看到内存使用的很少,才 32M,而 free 是 3G,cache_mem 是 1G。</p>
<p>还有,之后对全部日志进行分析时发现,cache_status 不单单是 <code class="highlighter-rouge">TCP_MEM_HIT</code> 会出现这种情况,全部日志的情况如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="o">[</span>root@tinysquid2 ~]# cat /cache/logs/access.log|grep HIT|grep DIRECT|grep -v REFRESH|awk <span class="s1">'{print $4}'</span>|sort|uniq -c
1086 TCP_HIT/200
18386 TCP_IMS_HIT/304
45964 TCP_MEM_HIT/200
2 TCP_MEM_HIT/206
</code></pre>
</div>
<p>即使是 DISK 上的 <code class="highlighter-rouge">TCP_HIT</code> 也有回源的。太奇怪了!!</p>
consistent_hash的perl脚本模拟
2010-03-16T00:00:00+08:00
web
perl
http://chenlinux.com/2010/03/16/implement-consistent_hash-by-perl
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/perl -w</span>
<span class="k">use</span> <span class="nn">String::</span><span class="nv">CRC32</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$url</span> <span class="o">=</span> <span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@peer</span> <span class="o">=</span> <span class="p">(</span><span class="s">'10.13.12.14:80'</span><span class="p">,</span>
<span class="s">'10.13.12.15:80'</span><span class="p">,</span>
<span class="s">'10.13.12.16:80'</span><span class="p">,</span>
<span class="s">'10.13.12.17:80'</span><span class="p">,</span>
<span class="s">'10.13.12.18:80'</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$new</span> <span class="o">=</span> <span class="mi">9999999999</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$sum</span> <span class="o">=</span> <span class="nv">$#peer</span><span class="o">+</span><span class="mi">1</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$uricrc</span> <span class="o">=</span> <span class="nv">crc32</span><span class="p">(</span><span class="s">"$url"</span><span class="p">,</span><span class="nb">length</span><span class="p">(</span><span class="nv">$url</span><span class="p">));</span>
<span class="k">for</span> <span class="p">(</span><span class="k">my</span> <span class="nv">$i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="nv">$i</span><span class="o"><</span><span class="nv">$sum</span><span class="p">;</span><span class="nv">$i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$peercrc</span> <span class="o">=</span> <span class="nv">crc32</span><span class="p">(</span><span class="nv">$peer</span><span class="p">[</span><span class="nv">$i</span><span class="p">],</span><span class="nb">length</span><span class="p">(</span><span class="nv">$peer</span><span class="p">[</span><span class="nv">$i</span><span class="p">]));</span>
<span class="k">my</span> <span class="nv">$res</span> <span class="o">=</span> <span class="nv">$peercrc</span> <span class="o">-</span> <span class="nv">$uricrc</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$res</span> <span class="o">></span> <span class="mi">0</span> <span class="o">&&</span> <span class="nv">$res</span> <span class="o"><</span> <span class="nv">$new</span><span class="p">){</span>
<span class="nv">$new</span> <span class="o">=</span> <span class="nv">$res</span><span class="p">;</span>
<span class="nv">$haha</span> <span class="o">=</span> <span class="s">"$i\t$new"</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">my</span> <span class="nv">@num</span> <span class="o">=</span> <span class="nb">split</span><span class="p">(</span><span class="sr">/\t/</span><span class="p">,</span><span class="nv">$haha</span><span class="p">);</span>
<span class="nb">printf</span><span class="p">(</span><span class="s">"%s cached at the %s peer by the key %010.0f.\n"</span><span class="p">,</span><span class="nv">$url</span><span class="p">,</span><span class="nv">$peer</span><span class="p">[</span><span class="nv">$num</span><span class="p">[</span><span class="mi">0</span><span class="p">]],</span><span class="nv">$uricrc</span><span class="p">);</span>
<span class="nv">_END_</span>
</code></pre>
</div>
<p>测试如下:<br />
<code class="highlighter-rouge">bash
./crc.pl http://www.hapi.com.cn/FLASH/age.swf
http://www.hapi.com.cn/FLASH/age.swf cached at the 10.13.12.14:80 peer by the key 1007905459.
</code><br />
方法很拙劣,在比较运算和数组传递之间捣腾了很久,最后还是没能用%var{}和sort的办法搞定,而是采用了赋值单一变量然后切割获取序号的办法,反正只是做个小模拟,~~</p>
一致性哈希研究
2010-03-15T00:00:00+08:00
web
nginx
perl
http://chenlinux.com/2010/03/15/consistent_hash
<p>今天继续看nginx的consistent_hash_module,因为想可能的话可以把url和对应的peer关系查出来,形成一个类似squidclient一样的方式。以下内容都是我从百度、谷歌、nginx模块应用指南和ngx_upstream_consistent_hash_module的src中自我理解得出的,欢迎指正。</p>
<h2 id="urlhash">一、url_hash的原理</h2>
<p>在一些稳定的系统(即不考虑流量变化导致时常add/del服务器)中,采用upstream_url_hash+固定的peer+一个backup的方式应该很不错。整个module很“简单”,对$uri进行CRC32计算得到key,然后把key针对peer的总数求余数,余几就把$uri存在第几个peer上。over~~</p>
<p>用perl表示,大概如下吧:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/perl -w</span>
<span class="k">use</span> <span class="nn">String::</span><span class="nv">CRC32</span><span class="p">;</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$url</span><span class="p">,</span> <span class="nv">$sum</span><span class="p">)</span> <span class="o">=</span> <span class="nv">@ARGV</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$crc</span> <span class="o">=</span> <span class="nv">crc32</span><span class="p">(</span><span class="s">"$url"</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$num</span> <span class="o">=</span> <span class="nv">$crc</span> <span class="nv">%</span> <span class="err">$</span><span class="nv">sum</span><span class="p">;</span>
<span class="nb">printf</span> <span class="s">"%s cached at the %s peer by the key %s\n"</span><span class="p">,</span> <span class="nv">$url</span><span class="p">,</span> <span class="nv">$num</span><span class="p">,</span> <span class="nv">$crc</span><span class="p">;</span>
</code></pre>
</div>
<p>测试如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">[</span>root@sdl4 /home/rao 21:41:47]# ./crc.pl http://www.baidu.com 10
http://www.baidu.com cached at the 4 peer by the key 3500265894
</code></pre>
</div>
<h2 id="consistenthash">二、consistent_hash的原理</h2>
<ol>
<li>consistent_hash在url_hash的基础上进一步,不单单对$uri进行CRC32计算,同样对peer进行CRC32计算。然后向下寻找离crc32($uri)最近的一个crc32($peer),并把$uri存在这个peer上。</li>
<li>单纯进行uri和peer的crc32并寻找最近点的话,在均衡方面做到并不好,因为事实上sum(peer)不大可能大到满足理论推测的,就这么五六台peer,肯定很容易就出现个别服务器爆满的情况,所以consistent_hash还有进阶做法,为真实的peer做虚拟节点,然后uri寻找最近的虚拟节点存储(当然实际上还是对应到真实peer了)。按照前人实验,如果10台peer的话,给每个peer分成100-200个虚拟节点,才能比较完美的达到url_balance。</li>
</ol>
<h2 id="ngxupstreamhashmodule">三、ngx_upstream_hash_module的原理</h2>
<p>nginx模块应用指南网上到处有,这里只贴CU论坛上关于upstream的一段翻译<a href="http://bbs.chinaunix.net/thread-1479873-1-1.html">http://bbs.chinaunix.net/thread-1479873-1-1.html</a>,大概说明的代码流程就是</p>
<ol>
<li>ngx_http_upstream_hash注册upstream初始化函数并填充CONF信息,即读取nginx.conf中upstream backend {}中的内容;</li>
<li>ngx_http_upstream_init_hash初始化函数,(如果peer不是IP是Domain的话)DNS resolv,(根据peer的数量、端口、权重等)allocate sockets;</li>
<li>ngx_http_upstream_peer_data_t初始化peer函数,计算hash,设置get、free、tries变量;</li>
<li>ngx_http_upstream_get_peer(ngx_peer_connection_t *pc, void *data),接受1中得到的peer列表,进行%运算,返回peer_name,告知nginx建立连接;</li>
<li>ngx_http_upstream_free_peer(ngx_peer_connection_t *pc, void *data, ngx_uint_t state),完成rehash。</li>
</ol>
<h2 id="ngxconsistenthashmodule">四、ngx_consistent_hash_module的原理</h2>
<p>这个module是在upstream的架构上完成的,所以对照上面的指南,倒是可以看出来一点点头绪。</p>
<ol>
<li>在计算crc32的时候,不单单是使用uri和server_name:port的字符串,而且还增补上了字符串length;</li>
<li>和url_hash禁止给server加其他任何配置不同,src中也有weight的相关定义(0-255)处理(计算虚拟节点时),以完成weight->hash_cyc的映射;具体算法如下,MMC_CONSISTENT_POINTS,最开始定义了它等于160.应该就是全默认状态下的虚拟节点,naddrs或许是当前server的排序序号?(未知)</li>
</ol>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="n">points</span> <span class="o">+=</span> <span class="n">server</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">weight</span> <span class="o">*</span> <span class="n">server</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">naddrs</span> <span class="o">*</span> <span class="n">MMC_CONSISTENT_POINTS</span><span class="p">;</span>
</code></pre>
</div>
<ol>
<li>ngx_http_upstream_consistent_hash_find(ngx_http_upstream_consistent_hash_continuum *continuum, ngx_uint_t point)函数test middle point,用来计算url的crc32离哪个point最近。</li>
<li>使用了ngx_crc32_long来计算hash,这部分在nginx/src/core/ngx_crc32.c中。看到里头提供了256的初始化数据,和perl的String::CRC32里的是一样的……</li>
</ol>
nginx泛域名cache_store
2010-03-13T00:00:00+08:00
nginx
http://chenlinux.com/2010/03/13/pan-domain-cache_store
<p>回到nginx的cache_store方式上来。这是传统的nginx缓存方式,配置一般如下:<br />
<code class="highlighter-rouge">nginx
upstream test{
server 211.152.60.180:80;
}
server {
listen 80;
server_name images6.static.com;
location / {
root /cache/images6.static.com/;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_store on;
proxy_store_access user:rw group:rw all:r;
proxy_temp_path /cache/temp;
if (!-f $request_filename) {
proxy_pass http://test;
}
}
}
</code><br />
很简单明了。不过如果如果碰上img[1-16].static.com这样的客户,难不成把这一大段复制粘贴上16遍?汗~~必然得采用泛域名方式了。</p>
<p>server_name 支持.static.com的方式,root也支持/cache/$host/没有问题,save&&reconfigure,wget试一下,却没能缓存住。</p>
<p>于是去翻nginx官方wiki,刚巧看到proxy_store语法,除了on和off这两个想当然的赋值外,还有一个path!原文如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Furthermore, the name of the path can be clearly assigned with the aid of the
line with the variables:
proxy_store /data/www$original_uri;
</code></pre>
</div>
<p>赶紧换上,测试果然成功!conf如下:<br />
<code class="highlighter-rouge">nginx
upstream test{
server 211.152.60.180:80;
}
server {
listen 80;
server_name .anjukestatic.com;
location / {
root /cache/$host/;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_store /cache/$host$uri;
proxy_store_access user:rw group:rw all:r;
proxy_temp_path /cache/temp;
if (!-f $request_filename) {
proxy_pass http://test;
}
}
}
</code><br />
测试日志记录如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1268456455.505 -/200 101 GET http://images6.static.com/property/20090911/600x600.jpg PARENT/211.152.60.180:80 "-" "Wget/1.10.2 (Red Hat modified)"
1268456586.371 -/200 101 GET http://images6.static.com/property/20090911/600x600.jpg PARENT/- "-" "Wget/1.10.2 (Red Hat modified)"
1268456631.414 -/200 45 GET http://images9.static.com/property/20100103/420x315.jpg PARENT/211.152.60.180:80 "-" "Wget/1.10.2 (Red Hat modified)"
1268456632.231 -/200 45 GET http://images9.static.com/property/20100103/420x315.jpg PARENT/- "-" "Wget/1.10.2 (Red Hat modified)"
</code></pre>
</div>
<p>不知道为什么,这个$upstream_cache_status居然一直是-,郁闷一下下。</p>
<p>另,刚开始只写proxy_store /cache/$host;也不行,后来看error.log中提示</p>
<div class="highlighter-rouge"><pre class="highlight"><code>“rename() "/cache/temp/0000000001" to "/cache/images6.static.com/" failed (20: Not a directory)”
</code></pre>
</div>
<p>才知道必须加上$uri。官方文档写的是$original_uri,日志里写的是$request_uri,nginx的内置变量有时候真的让人有些头晕……<br />
思路跳回上篇的大小写,或许用下面这个办法可以?<br />
<code class="highlighter-rouge">nginx
perl_set $url '
sub {
my $r = shift;
my $re = lc($r->uri);
return $re;
}
';
proxy_store /cache/$host$url;
#proxy_cache_key $host$url$is_args$args;
</code><br />
未经试验,目前猜测,可能结果是nginx回源下载新文件,然后覆盖掉原来的——也就是说达到节省磁盘的目的,但HIT/MISS照旧。</p>
忽略大小写(nginx)
2010-03-13T00:00:00+08:00
nginx
http://chenlinux.com/2010/03/13/ignore-case-in-nginx
<p>刚才发现使用perl_set忽略大小写,完全不用perl_module和perl_require那么兴师动众,同样也能达到不错的效果。比如<a href="http://www.cnblogs.com/fengmk2/archive/2009/04/25.html" target="_blank">这个用perl做伪静态路径的例子</a>。随即就动手试验一下。</p>
<p>首先找一个windows的origin,因为windows是不区分大小写,这样可以确保任意wget都能返回200的结果;</p>
<p>然后按照上篇提到的方法配置nginx.conf(如下),stop&&start看看。<br />
<code class="highlighter-rouge">nginx
upstream test{
server 61.152.237.170:80;
}
perl_set $url '
sub {
my $r = shift;
my $re = lc($r->uri);
return $re;
}
';
server {
listen 80;
server_name www.hapi.com.cn;
location / {
root /cache/$host/;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_store /cache/$host$url;
proxy_store_access user:rw group:rw all:r;
proxy_temp_path /cache/temp;
if (!-f $request_filename) {
proxy_pass http://test;
}
}
}
</code><br />
这里没法直接把lc($uri)继续set成$uri,应该是内置变量的缘故……</p>
<p>测试相关的access.log如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1268461739.707 -/200 416073 GET http://www.hapi.com.cn/flash/age.swf PARENT/61.152.237.170:80 "-" "Wget/1.10.2 (Red Hat modified)"
1268461767.366 -/200 416073 GET http://www.hapi.com.cn/flash/Age.swf PARENT/61.152.237.170:80 "-" "Wget/1.10.2 (Red Hat modified)"
1268461785.360 -/200 416073 GET http://www.hapi.com.cn/flash/Age.swf PARENT/61.152.237.170:80 "-" "Wget/1.10.2 (Red Hat modified)"
1268461806.195 -/200 416073 GET http://www.hapi.com.cn/flash/age.swf PARENT/- "-" "Wget/1.10.2 (Red Hat modified)"
</code></pre>
</div>
<p>然后查看缓存目录:</p>
<p>[root@sdl4 /home/nginx/conf 14:36:46]# ls /cache/www.hapi.com.cn/flash/<br />
age.swf</p>
<p>可以看到,只缓存了一个文件,但其他写法的请求就会反复重写……</p>
<p>看起来缓存空间确实是节省下来了,不过真正的缓存目的还是没达到。</p>
<p>再加上rewrite,变成下面这样:<br />
<code class="highlighter-rouge">nginx
perl_set $url '
sub {
my $r = shift;
my $re = lc($r->uri);
return $re;
}
';
server {
listen 80;
server_name www.hapi.com.cn;
if ($uri ~ [A-Z]){
rewrite ^(.*)$ $url last;
}
location / {
root /cache/$host/;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_store /cache/$host$uri;
proxy_store_access user:rw group:rw all:r;
proxy_temp_path /cache/temp;
if (!-f $request_filename) {
proxy_pass http://test;
}
}
}
</code><br />
测试变可以了:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1268469316.852 -/200 416073 GET http://www.hapi.com.cn/flash/agE.swf PARENT/- "-" "Wget/1.10.2 (Red Hat modified)"
1268469327.605 -/200 416073 GET http://www.hapi.com.cn/FLASH/age.swf PARENT/- "-" "Wget/1.10.2 (Red Hat modified)"
1268469397.312 -/200 416073 GET http://www.hapi.com.cn/FLASH/AGE.swf PARENT/- "-" "Wget/1.10.2 (Red Hat modified)"
</code></pre>
</div>
<p>另:因为uri改写后,是从location开始重新执行匹配等(相当于重新访问),所以这里proxy_store用$uri就行了——换句话说,前面那一大段都是白折腾。。。</p>
忽略大小写(刚在nginx的maillist看到的)
2010-03-13T00:00:00+08:00
nginx
http://chenlinux.com/2010/03/13/ignore-case-in-nginx-2
<p>想找找解决不同client请求中文url产生多份cache的办法,结果在nginx的maillist里看到也有人问《<a href="http://forum.nginx.org/read.php?2,48527" target="_blank">rewrite to lowercase?</a>》下面有人回答了这个问题,做法和我一样~~呵呵。随后他也附上了这种做法的思路和问题。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>The first option (with the if and http://$host) sends an HTTP redirect
to the lowercase URL if the requested URL is not completely in
lowercase. I like this approach better as it kind of normalizes all URL
to a canonical form.. The second option (without if nor http://) just accepts
any URL and internally rewrites them to lowercase: if you rename all
your directories and files to lowercase, this will imitate Windows'
case-insensitive behaviour.
</code></pre>
</div>
<p>第一个办法:采用if语句判断url是否有大写字母在内,有则重定向,作者本人比较喜欢这种方式,因为这样返回的url是符合标准规范的;第二个方法,采用internal方式重定向所有uri,就像windows主机的做法一样。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Kudos to the following post were I
draw the "inspiration" :D from to get this done. The embedded perl doc
and examples are indeed scarce. :-(
</code></pre>
</div>
<p>最后作者也感慨了一句:nginx的内置perl模块的文档和配置案例实在是太少了……<br />
Anyway I see two problems in this approach:<br />
然后说说这种办法的两个问题:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>- you need the
embedded perl module which according to the docs is experimental and can
lead to memory leaks.
</code></pre>
</div>
<p>这个方法是基于内嵌perl模块的,而根据官方文档的说明,有可能导致内存泄漏!</p>
<div class="highlighter-rouge"><pre class="highlight"><code>- the actual redirection is done with the
rewrite, which you can put on the location you need. But the URL
lowercase calculation, being on the "http" section of the config, is
done for each and every request arriving to your server. Say you have a
virtual server with 10 domains and you only need this on one particular
location of one of them. The lowercase URL is going to be calculated for
every request of every domain.
</code></pre>
</div>
<p>rewrite语句是在location段的,而perl_set语句是在http段的。也就是说,每一个请求过来,都被转化大小写了。如果你的服务器上配了十个虚拟主机,却只想忽略一个的大小写,这个换算依然要在所有域名中都执行的……</p>
<div class="highlighter-rouge"><pre class="highlight"><code>I guess that by defining a perl
function the URL calculation could be restricted to a particular
location, have to look into it further. Another option is writing a C
module.
</code></pre>
</div>
<p>我想在location段里另建perl函数换算url(即perl_module/perl_require),不过这是以后的事了;或者干脆用C语言写个模块。</p>
<p>在继续往下翻的时候,看到两条感兴趣的话,一是nginx开发者正在完成updating_file_lock的功能,解决第一次MISS的时候同时向origin并发请求的问题;二是正在完善cache_path和temp_pache必须在同一个文件系统的link()的功能,这样就可以在每个server下指定特定的cache_path了。</p>
nginx日志(upstream)
2010-03-11T00:00:00+08:00
nginx
http://chenlinux.com/2010/03/11/upstream-log-in-nginx
<p>作为一个web服务器,我们已经习惯了nginx的类apache日志,即$status。其实nginx的upstream模块下,带有几个变量,却是类squid日志的。他们是:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$upstream_addr
$upstream_cache_status
$upstream_status
</code></pre>
</div>
<p>当log_format main ‘$msec ‘‘$remote_addr ‘‘$upstream_cache_status/$upstream_status ‘‘$body_bytes_sent ‘‘$request_method ‘‘$scheme://$http_host$request_uri ‘‘$remote_user ‘‘$upstream_addr ‘’“$http_referer” ‘’“$http_user_agent” ‘;的时候,访问一个可cache的url的log结果如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1268324542.656 127.0.0.1 MISS/200 60 GET http://flv.91091.net/skins/meihong/images/icon_album.gif - 202.102.79.133:80 "-" "Wget/1.10.2 (Red Hat modified)"
1268324544.505 127.0.0.1 HIT/- 60 GET http://flv.91091.net/skins/meihong/images/icon_album.gif - "-" "Wget/1.10.2 (Red Hat modified)"
</code></pre>
</div>
<p>$upstream_cache_status除了MISS和HIT外,还有EXPIRED、UPDATING和STALE三个赋值。这三个赋值的原文解释如下。</p>
<p>** EXPIRED - expired, request was passed to backend<br />
** UPDATING - expired, stale response was used due to proxy/fastcgi_cache_use_stale updating<br />
** STALE - expired, stale response was used due to proxy/fastcgi_cache_use_stale</p>
<p>这个STALE应该类似TCP_REFRESH_HIT,EXPIRED是TCP_MISS和TCP_REFRESH_MISS,UPDATING有些不好对比了。这个可能跟squid和nginx的传输方式不太同有关。squid对数据,是边从oringin拿边传给client的,而nginx要全部拿完才给。这期间,应该就是UPDATING??</p>
忽略大小写(nginx|apache)
2010-03-11T00:00:00+08:00
nginx
nginx
apache
perl
http://chenlinux.com/2010/03/11/ignore-case-in-nginx-apache
<p>看了看nginx的perl_module,差不多知道了个大概。<br />
在nginx.conf的http域中,通过perl_modules指定模块路径,perl_require指定模块名称;location域中通过perl引用函数。<br />
引用对象为$r->**,列举一下主要的参数:<br />
·$r->args - 返回请求的参数。<br />
·$r->discard_request_body - 告诉nginx忽略请求主体。<br />
·$r->filename - 更具URI的请求返回文件名。<br />
$r->has_request_body(function) - 如果没有请求主体,返回0,但是如果请求主体存在,那么建立传递的函数并返回1,在程序的最后,nginx将调用指定的处理器。<br />
·$r->header_in(header) - 检索一个HTTP请求头。<br />
·$r->header_only - 在我们只需要返回一个应答头时为真。<br />
·$r->header_out(header, value) - 设置一个应答头。<br />
·$r->internal_redirect(uri) - 使内部重定向到指定的URI,重定向仅在完成perl脚本后发生。<br />
·$r->print(args, …) - 为客户端传送数据。<br />
·$r->request_body - 在请求主体未记录到一个临时文件时为客户返回这个请求主体。为了使客户端的请求主体保证在内存里,可以使用client_max_body_size限制它的大小并且为其使用的缓冲区指定足够的空间。<br />
·$r->request_body_file - 返回存储客户端需求主体的文件名,这个文件必须在请求完成后被删除,以便请求主体始终能写入文件,需要指定client_body_in_file_only为on。<br />
·$r->request_method - 返回请求的HTTP动作。<br />
·$r->remote_addr - 返回客户端的IP地址。<br />
·$r->rflush - 立即传送数据到客户端。<br />
·$r->sendfile(file [, displacement [, length ] ) - 传送给客户端指定文件的内容,可选的参数表明只传送数据的偏移量与长度,精确的传递仅在perl脚本执行完毕后生效。<br />
·$r->send_http_header(type) - 为应答增加头部,可选参数“type”在应答标题中确定Content-Type的值。<br />
·$r->sleep(milliseconds, handler) - 设置为请求在指定的时间使用指定的处理方法和停止处理,在此期间nginx将继续处理其他的请求,超过指定的时间后,nginx将运行安装的处理方法,注意你需要为处理方法通过一个reference,在处理器间转发数据你可以使用$r->variable()。<br />
·$r->status(code) - 设置HTTP应答代码。<br />
·$r->unescape(text) - 以%XX的形式编码text。<br />
·$r->uri - 返回请求的URI。<br />
·$r->variable(name[, value]) - 返回一个指定变量的值,变量为每个查询的局部变量。</p>
<p>nginx本身关于该模块的例子不多。除了官网<a href="http://www.freebsdsystem.org/doc/nginx_zh/OptionalHTTPmodules/EmbeddedPerl.html" target="_blank">三个用法举例</a>外,我只在一个博客上看到另<a href="http://hi.baidu.com/ywdblog/blog/item/172010d1c8de0dd5572c8487.html" target="_blank">一个用perl记录log的例子</a>。好在apache的perl例子很多,大约可以参见一下:<br />
在<a href="http://perl.apache.org/docs/1.0/guide/snippets.html" target="_blank">mod_perl:Code Snippets</a>中关于mod_rewrite的两个举例:<br />
<code class="highlighter-rouge">perl
package Apache::MyRedirect;
use Apache::Constants qw(OK REDIRECT);
use constant DEFAULT_URI => 'http://www.example.org';
sub handler {
my $r = shift;
my %args = $r->args;
my $path = $r->uri;
my $uri = (($args{'uri'}) ? $args{'uri'} : DEFAULT_URI) . $path;
$r->header_out(Location => $uri);
$r->status(REDIRECT);
$r->send_http_header;
return OK;
}
_END_
package My::Trans;
use Apache::Constants qw(:common);
sub handler {
my $r = shift;
my $uri = $r->uri;
my ($id) = ($uri =~ m|^/articles/(.*?)/|);
$r->uri("/articles/index.html");
$r->args("id=$id");
return DECLINED;
}
1;
_END_
</code><br />
而在<a href="http://book.opensourceproject.org.cn/lamp/perl/perlcook2/index.html?page=opensource/0596003137_perlckbk2-chp-21-sect-4.html" target="_blank">perl cookbook</a>中的21.4节redirecting the browser中,则说明了perl模块的两种转向方法及其流程的不同: <br />
$r->header_out(Location => “http://www.example.com/somewhere”); <br />
return REDIRECT; <br />
这个方法,是要把新url返回给browser,由browser端再发出DNS解析等一系列请求活动; <br />
$r->internal_redirect($new_partial_url); <br />
return OK; <br />
这个方法,是server自己内部重定向,虽然apache本身对这个请求依然要从头开始走一遍流程。而且这里的新url,只需要提供一个相对路径——唯一要注意的,是这个redirect之后不能再进行逻辑判断了,最好直接返回OK。</p>
忽略大小写(squid)
2010-03-09T00:00:00+08:00
squid
squid
perl
http://chenlinux.com/2010/03/09/ignore-case-in-squid
<p>在配置squid.conf的refresh_pattern或者url_regex的时候,我们习惯性的都会加上一个options:“-i”,用于忽略大小写。<br />
前不久配置nginx.conf的location时,也用上了~*忽略大小写。<br />
但是这个“忽略大小写”,其实只是整个请求处理流程中部分过程——配置规则的匹配过程——中的忽略。<br />
在使用了这些options以后,一个http://www.test.com/a.htm和另一个http://www.test.com/A.HTM请求在到达squid/nginx的时候,会统一无视大小写的进行规则匹配,然后可能proxy_pass到oringin获取数据;接下来有两种情况:<br />
oringin是windows主机,不区分大小写,返回200数据,缓存下来——分别是a和A两份!<br />
oringin是类unix主机,区分大小写,返回一个正确的200,一个错误的404……甚至可能两个都404~~<br />
(根据测试,完整的url中,host是不用区分大小写的,url_path里的大小写才有影响)<br />
如果要让同样的内容就缓存一份数据,我想就只能在squid/nginx内核在处理url之前,将url的大小写问题处理掉。</p>
<p>squid可以url_rewrite_program,perl很简单的lc()即可,如下:<br />
<code class="highlighter-rouge">perl
#!/usr/bin/perl -w
$|=1;
while(<>){
my @X=split;
my $uri=lc($X[0]);
print($uri);
}
</code></p>
<p>nginx的http_perl_module,看起来像是做这个的,不过官网上声明说,里面的perl如果执行太多的查询类操作(比如DNS域名解析、数据库操作等),很可能就把nginx的worker-process跑挂了……;且启用这个模块的话,就不能reconfigure,否则可能内存溢出云云……</p>
ASUS-EeePC-1005HA
2010-03-08T00:00:00+08:00
http://chenlinux.com/2010/03/08/asus-eeepc-1005ha
<p>去年参加华硕服务器平台搭建大赛得了二等奖,延误了这么久,终于把奖品拿到手了—-一台上网本.赶紧开始试用.</p>
<p>开机,发现是linux系统,而这个界面看起来就象是手机一样….</p>
<p>习惯性的按下ctrl+alt+F1,汗,居然没有反应!—-我在网上翻了几个小时,才算偶然看到原来这个怪系统里只能用ctrl+alt+t调出虚拟终端.其他的东东已经都被华硕删掉了……</p>
<p>决定升级,已经知道了这个系统是基于debian的,赶紧换上debian的中科大源,居然失败;换debian官方源,一样不行……等我好不容易找到一个xandros的源,居然一样不能用—-因为没有pubkey.我又一路google到EeeUser论坛,看别人如何如何打key,照做之后,还是error……</p>
<p>打开终端看看ps aux,赫然发现一个nginx,路径是杀毒软件ESET下,版本号是0.5.*.看nginx.conf,开了一个诡异的20032端口.我汗,预装linux也不忘杀毒软件呀~</p>
<p>上网打字,发现这个输入法更奇怪,用空格翻页,按住shift输英文,词组不全咱都不怪它,问题是词组选择编号居然是跳跃性的.看来搜狗云输入法真是及时雨……</p>
<p>正上方是一个摄像头,问题是:中国用skype视频聊天的人与用qq视频聊天的人完全是不成比例呀…qq可没法在linux下用视频….</p>
<p>散热真的超级小, 不知道改天换成windows后如何……<br />
要是没有qq聊天视频的需求,或许我真就干脆这么用下去得了~~~</p>
awk收发邮件小脚本
2010-03-07T00:00:00+08:00
bash
awk
http://chenlinux.com/2010/03/07/send-and-receive-mail-by-awk
<p>前两天做监控的小脚本,其实只是采集几个数据,再调用了一个现成的perl大脚本发送出来,最后归入了monitor类别,没好意思放进shell类别里。</p>
<p>今天在CU上翻老帖,看到一个awk实现的收邮件方法。对POP3,对awk,都是很让人开眼界的。在原有基础上,针对性的略加改动,也就完成了SMTP的发邮件awk脚本。这回,可以光明正大的放进shell类别里了~~</p>
<p>收邮件的脚本:<br />
<code class="highlighter-rouge">bash
#!/usr/bin/gawk -f
BEGIN {
#gawk调用socket的网络通信格式:/inet/<tcp|udp|raw>/<0|local_port>/<remote_host>/<remote_port>
Service = "/inet/tcp/0/mail.test.com/110"
#gawk从ksh借鉴来的双向管道
Service |& getline;
print;
print "USER raocl" |& Service
Service |& getline;
print;
print "PASS raocl" |& Service
Service |& getline;
print;
print "LIST" |& Service
while ((Service |& getline line) > 0 && !(line~/^./) && ++mailCount) print line;
mailCount--;
if (mailCount > 0){
print "RETR "mailCount |& Service
while ((Service |& getline line) > 0 && !(line~/^./)) print line;
}
print "QUIT" |& Service
close(Service)
}
</code><br />
在POP3中,还可以使用TOP n m来获取data的前几行,NOOP来保持与server的连接,DELE来删除邮件。<br />
发邮件的脚本:<br />
```bash<br />
#!/usr/bin/gawk -f<br />
BEGIN {<br />
Service = “/inet/tcp/0/mail.test.com/25”<br />
Service |& getline;<br />
print;<br />
print “HELO mail.test.com” |& Service<br />
Service |& getline;<br />
print;<br />
print “MAIL FROM: user1@mail.test.com” |& Service<br />
Service |& getline;<br />
print;<br />
print “RCPT TO: user2@mail.test.com” |& Service<br />
Service |& getline;<br />
print;<br />
print “data” |& Service</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Service |& getline;
print;
print "from: user1@mail.test.com" |& Service
print "to: <a href="mailto:user2@mail.test.com">user2@mail.test.com</a>" |& Service
print "date: Fir, 8 Mar 2010 01:11:00 +0800" |& Service
print "subject: testmail" |& Service
print "hello, world!" |& Service
print "." |& Service
print;
print "QUIT" |& Service
close(Service) } ``` 好了,运行一下吧。
# ./sendmail.awk && ./getmail.awk
220 mail.test.com ESMTP jmcs.mta (2.2.2)
250 mail.test.com
250 Ok
250 Ok
354 End data with <CR><LF>.<CR><LF>
354 End data with <CR><LF>.<CR><LF>
+OK scan listing follows
1 820
2 1765
3 1777
4 1765
5 768
+OK Message follows
Return-Path: <<a href="mailto:user1@mail.test.com">user1@mail.test.com</a>>
Received: from mail.test.com ([unix socket])
by mail.21vianet.com (JMessage v2.3) with LMTP; Mon, 08 Mar 2010 01:14:43 +0800
X-Sieve: CMU Sieve 2.2
Received: from jmcs.antivirus (localhost.localdomain [127.0.0.1])
by mail.test.com (jmcs.mta) with SMTP id B7268B2EC4E0
for <<a href="mailto:user2@mail.test.com">user2@mail.test.com</a>>; Mon, 8 Mar 2010 01:14:43 +0800 (CST)
Received: from mail.test.com (unknown [218.60.36.39])
by mail.test.com (jmcs.mta) with SMTP id A38B3B2EC4DE
for <<a href="mailto:user2@mail.test.com">user2@mail.test.com</a>>; Mon, 8 Mar 2010 01:14:43 +0800 (CST)
from: <a href="mailto:user1@mail.test.com">user1@mail.test.com</a>
to: <a href="mailto:user2@mail.test.com">user2@mail.test.com</a>
date: Fir, 8 Mar 2010 01:11:00 +0800
subject: testmail
Message-Id: <<a href="mailto:20100307171443.A38B3B2EC4DE@mail.test.com">20100307171443.A38B3B2EC4DE@mail.test.com</a>>
hello, world!
</code></pre>
</div>
<p>就是这样~~报警的话,再继续插入变量处理就行了。在上面的getmail.awk中,其实我和CU上原帖唯一的不同就是他取固定的第7封,而我采用变量取固定的最新一封。这个getmail.awk,下一步可以自动获取信件内容中的IP和alarm-value,然后以cgi的方式显示在网页上(只用一个FF,少开一个TB~自汗一个)</p>
<p>另,awk调用外部变量不方便,也可以采用expect脚本spawn telnet的方式进行自动交互。</p>
<p>其实本来是室友部门部署监控,嫌cacti的报警不像nagios那么醒目了然。我想thold插件既然都能发出mail报警,只要把报警的data变量内容截下来写进一个页面的table里,修改bgcolor显示red不就好了?可是万恶的屏蔽字眼居然把cactiusers.org也给干掉了……没地方下cacti插件了~哭。。。</p>
nginx的proxy_cache和cache_purge模块试用记录
2010-03-07T00:00:00+08:00
nginx
http://chenlinux.com/2010/03/07/intro-proxy_cache-ngx_cache_purge
<p>nginx的类squid哈希式cache功能,据张宴说是基本稳定可用了,昨天找个机会和时间,试着测用了一把,把要点记录一下:</p>
<p>首先是编译nginx,方便起见,把一些心仪的模块统统加上了,version如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>built by gcc 4.1.2 20080704 (Red Hat 4.1.2-44)
TLS SNI support disabled
configure arguments: --prefix=/home/nginx --with-pcre --add-module=../ngx_http_consistent_hash --add-module=../ngx_max_connections --add-module=../ngx_cache_purge --add-module=../ngx_mp4_streaming_public --with-cc-opt=-O3 --with-http_stub_status_module --with-http_ssl_module --with-http_flv_module --without-http_memcached_module --without-http_fastcgi_module --with-google_perftools_module
</code></pre>
</div>
<p>编译过程中几个注意事项:</p>
<ol>
<li>必须采用–with-pcre,而不要偷懒采用–without-http_rewrite_module,否则配置文件里将不支持if判断;</li>
<li>加载mp4_streaming,必须采用–with-cc-opt=-O3方式进行编译。</li>
<li>max_connections模块默认支持nginx最新版本是0.8.32,需要vi修改其DIR,然后path -p0,但千万不要看见有个Makefile就执行make && make install了,因为它会毫无道理的把整个nginx安装到当前目录的.nginx下隐藏起来……</li>
</ol>
<p>话说我add这个max_connections模块能怎么用自己也没想好,反正官方有limit_zone和limit_req限制client,再add个限制nginx2oringin的也不在乎吧……汗~~</p>
<p>比较囧的一点是,经过我折腾的nginx,虽然去除了debug -g模式编译,还是有4M多大……</p>
<p>sina的ncache模块,在我下载的最新的nginx0.8.34src上无法使用,而且ncache作者介绍说ncache的缓存不用内存,且其purge方式为标记为过期但并不更改文件内容直到下次访问请求以节省磁盘IO的负担;但根据我的试验,nginx的cache_purge模块则是采用了删除过期文件的方式进行(当然,proxy_cache的过期还是标记而不删除的,不然太耗IO了……)。</p>
<p>然后贴一下,实验完成后的配置文件:<br />
```nginx<br />
user nobody nobody;<br />
worker_processes 1;<br />
google_perftools_profiles /tmp/tcmalloc;<br />
worker_rlimit_nofile 65535;</p>
<p>events<br />
{<br />
use epoll;<br />
worker_connections 65535;<br />
}</p>
<p>http<br />
{<br />
include mime.types;<br />
default_type application/octet-stream;</p>
<div class="highlighter-rouge"><pre class="highlight"><code>log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log logs/access.log main ;
# charset utf-8;
server_names_hash_bucket_size 128;
client_header_buffer_size 32k;
large_client_header_buffers 4 32k;
client_max_body_size 300m;
sendfile on;
tcp_nopush on;
keepalive_timeout 60;
tcp_nodelay on;
client_body_buffer_size 512k;
proxy_connect_timeout 5;
proxy_read_timeout 60;
proxy_send_timeout 5;
proxy_buffer_size 16k;
proxy_buffers 4 64k;
proxy_busy_buffers_size 128k;
proxy_temp_file_write_size 128k;
gzip on;
gzip_min_length 1k;
gzip_buffers 4 16k;
gzip_http_version 1.1;
gzip_comp_level 2;
gzip_types text/plain application/x-javascript text/css application/xml;
gzip_vary on;
#定义cache临时缓存路径,必须和哈希缓存路径在同一个磁盘上
proxy_temp_path /cache/proxy_temp_dir;
#定义cache哈希缓存路径,目录层次,缓存名称及所允许缓存的最大文件大小,未被访问文件多久自动清除,缓存最多使用多大磁盘
proxy_cache_path /cache/proxy_cache_dir levels=1:2 keys_zone=cache_one:200m inactive=1d max_size=30g;
#后端源站地址
upstream backend{
server 10.10.10.13:80;
}
server
{
listen 80;
server_name www.test.com;
#这里就是关键部分了,定义哈希缓存及缓存过期;
#因为nginx提供的过期控制是针对http_status_code的,我本想通过location中限定类型的方法完成曲线救国,结果发现:一旦location中限定了文件类型,缓存过期的定义就失效!!
#也就是说,限定文件类型后的哈希缓存,是绝绝对对的强制永久缓存——不单过期失效,下面的purge也失效——或许换一个场景,这个刚好有用。
# location ~* .*.(css|gif|jpg|png|html|swf|flv)
location / {
#启用flv拖动功能;
flv;
#这里定义是如果碰上502、504、timeout和invalid_header等情况,自动调向下一个oringin继续请求,这个有时候有用,有时候可能被攻击的会很惨……
proxy_next_upstream http_502 http_504 error timeout invalid_header;
#使用上面定义的具体某个缓存;
proxy_cache cache_one;
#200和304的状态码访问都缓存1天;
proxy_cache_valid 200 304 1d;
#由主机名、唯一资源定位符、参数判断符和请求参数共同生成哈希缓存的key
proxy_cache_key $host$uri$is_args$args;
#下面几个是常见的nginx透明代理header
proxy_pass_header User-Agent;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
#下面是为了证明给大家看他确实能缓存,增加的两句话;如果真想要看到HIT和MISS的话,可以addnginx的另一个模块slowfs_cache,配置上和官方的proxy_cache极其相似,不过自带有变量$slowfs_cache_status,可以显示HIT/MISS/EXPIRED。
add_header X-Cache "HIT from cache_test";
add_header Age "1";
proxy_pass http://backend;
}
#下面这段就是add的purge_mod,只要在url的^/前再加上/purge,就会自动被理解成PURGE请求,刷新成功返回200的特定页面,失败返回404的普通错误页面。
#这个proxy_cache_purge格式没法改变,我本想改成if ($request_method = PUREG){…}试试,结果发现它不认……
location ~ /purge(/.*){
allow 127.0.0.1;
allow 211.151.67.0/24;
deny all;
proxy_cache_purge cache_one $host$1$is_args$args;
}
#不区分大小写匹配~*,网上流传很广的写法*~*是错滴;而且能匹配不代表保存同一份缓存;
#这部分定义不缓存而是透传的请求类型。介于无法通过类型来控制缓存,那么这里不缓存的控制就必须确保严格正确了……
#可是bug来了,当我写成swf?$的时候,swf/swf?/swf?*都不缓存;写成swf?的时候,又变成都缓存——nginx压根就分不清!
location ~* .*.(swf|asp)?{
proxy_pass_header User-Agent;
proxy_set_header Host $host;
proxy_set_header X-Forwarder-For $remote_addr;
add_header X-Cache "MISS from cache_test";
proxy_pass http://backend;
}
} } ``` 配置完成。测试如下: wget http://www.test.com/List/j.Html -S -O /dev/null -e http_proxy=127.0.0.1
--13:17:48-- http://www.test.com/List/j.Html
Connecting to 127.0.0.1:80... connected.
Proxy request sent, awaiting response...
HTTP/1.1 200 OK
Server: squid/2.6.STABLE21
Date: Sun, 07 Mar 2010 05:17:48 GMT
Content-Type: text/html; charset=utf-8
Connection: close
Vary: Accept-Encoding
Content-Length: 25126
Last-Modified: Wed, 10 Feb 2010 09:38:26 GMT
ETag: "948acecb34aaca1:6647"
X-Powered-By: ASP.NET
X-Cache: HIT from cache_test
Age: 1
Accept-Ranges: bytes
Length: 25126 (25K) [text/html]
Saving to: `/dev/null'
100%[====================================================================================================================>] 25,126 --.-K/s in 0s
</code></pre>
</div>
<p>13:17:48 (521 MB/s) - `/dev/null’ saved [25126/25126]</p>
<p>[root@sdl4 ~ 13:17:48]#<br />
wget -S -O /dev/null -e http_proxy=127.0.0.1 “http://www.test.com/Search.ASP?KeyWord=整形视频”<br />
–13:17:50– http://www.test.com/Search.ASP?KeyWord=%D5%FB%D0%CE%CA%D3%C6%B5<br />
Connecting to 127.0.0.1:80… connected.<br />
Proxy request sent, awaiting response…<br />
HTTP/1.1 200 OK<br />
Server: squid/2.6.STABLE21<br />
Date: Sun, 07 Mar 2010 05:17:51 GMT<br />
Content-Type: text/html; charset=utf-8<br />
Connection: close<br />
Vary: Accept-Encoding<br />
X-Powered-By: ASP.NET<br />
Content-Length: 24987<br />
Set-Cookie: ASPSESSIONIDSSQQSBST=KDPDLODBNLGONONBNHCJCEGP; path=/<br />
Cache-control: private<br />
X-Cache: MISS from cache_test<br />
Length: 24987 (24K) [text/html]<br />
Saving to: `/dev/null’</p>
<div class="highlighter-rouge"><pre class="highlight"><code>100%[====================================================================================================================>] 24,987 25.6K/s in 1.0s
13:17:52 (25.6 KB/s) - `/dev/null' saved [24987/24987] 完成。 至于?的问题,目前针对需要,倒有另一个办法: 在location / {}中,根据请求参数判断进行传递。即写成如下: ```nginx location / {
……
if ($is_args){
add_header X-Cache "MISS from cache_test";
proxy_pass http://backend;
} } location ~* .*\.asp{
proxy_pass_header User-Agent;
proxy_set_header Host $host;
proxy_set_header X-Forwarder-For $remote_addr;
add_header X-Cache "MISS from cache_test";
proxy_pass http://backend; } ``` 不过依然有问题:
</code></pre>
</div>
<ol>
<li>nginx的if不支持&&或者||,万一有些类型(比如htm和jpg)又要求带?的也缓存,显然又和这个$is_args冲突;<br />
或许采用下面的办法能继续区分?(未试验)<br />
<code class="highlighter-rouge">nginx
set $yn $is_args;
if ($uri ~* .(htm|jpg)){
set $yn "";
}
if ($yn){
proxy_pass http://backend;
}
</code><br />
;</li>
<li>nginx的if中不单不支持proxy_cache,居然也不支持proxy_set_header等定义,只能单纯的proxy_pass。</li>
</ol>
nginx负载均衡(consistent_hash、error_page)进阶
2010-03-03T00:00:00+08:00
nginx
http://chenlinux.com/2010/03/03/loadbalance-in-nginx-using-consistent_hash-error_page
<p>话接上篇,采用consistent方式进行url_hash负载均衡。</p>
<p>重新编译nginx过程如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget http://download.github.com/replay-ngx_http_consistent_hash-77b6940.tar.gz
tar zxvf replay-ngx_http_consistent_hash-77b6940.tar.gz
<span class="nb">cd </span>nginx-0.7.65
./configure --prefix<span class="o">=</span>/home/nginx --with-pcre<span class="o">=</span>/tmp/pcre-8.01 --with-http_stub_status_module --with-http_ssl_module --without-http_rewrite_module --add-module<span class="o">=</span>/tmp/nginx_upstream_hash-0.3 --add-module<span class="o">=</span>/tmp/replay-ngx_http_consistent_hash-77b6940
make <span class="o">&&</span> make install
</code></pre>
</div>
<p>完成。配置文件修改如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">upstream</span> <span class="s">images6.static.com</span> <span class="p">{</span>
<span class="kn">server</span> <span class="nf">11.11.11.12</span><span class="p">:</span><span class="mi">80</span><span class="p">;</span>
<span class="kn">server</span> <span class="nf">11.11.11.13</span><span class="p">:</span><span class="mi">80</span><span class="p">;</span>
<span class="kn">consistent_hash</span> <span class="nv">$request_uri</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
<p>访问测试正常。</p>
<p>error_page配置备份如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">upstream</span> <span class="s">backup</span> <span class="p">{</span>
<span class="kn">server</span> <span class="nf">11.11.11.14</span><span class="p">:</span><span class="mi">80</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">error_page</span> <span class="mi">404</span> <span class="mi">500</span> <span class="mi">502</span> <span class="mi">503</span> <span class="mi">504</span> <span class="p">=</span><span class="mi">200</span> <span class="s">@fetch</span><span class="p">;</span>
<span class="k">location</span> <span class="s">@fetch</span> <span class="p">{</span>
<span class="kn">proxy_pass </span> <span class="s">http://backup</span><span class="p">;</span>
<span class="kn">proxy_set_header </span> <span class="s">Host </span> <span class="nv">$host</span><span class="p">;</span>
<span class="kn">proxy_set_header </span> <span class="s">X-Real-IP </span> <span class="nv">$remote_addr</span><span class="p">;</span>
<span class="kn">proxy_set_header </span> <span class="s">X-Forwarded-For </span> <span class="nv">$proxy_add_x_forwarded_for</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
<p>注意:=200里的等号,左边有空格,右边没空格。</p>
服务器监控报警小脚本(shell+sendEmail)
2010-03-03T00:00:00+08:00
monitor
bash
perl
http://chenlinux.com/2010/03/03/an-alarm-shell-using-sendemail
<p>这种email报警脚本遍地都是,很多用的sendmail、postfix,感觉有些大材小用了;也有些用perl的NET::SMTP和Authen::SASL模块发信的,不过我perl用的不好,老发出些莫名其妙的邮件来(比如if(a>1){print(a);},最后邮件里的显示的是0.99……);最后采用sendEmail这个成型的perl程序发信报警,而实时监控部分回归shell,终于完成。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>wget <a <span class="nv">href</span><span class="o">=</span><span class="s2">"http://caspian.dotconf.net/menu/Software/SendEmail/sendEmail-v1.56.tar.gz"</span>>http://caspian.dotconf.net/menu/Software/SendEmail/sendEmail-v1.56.tar.gz</a>
tar zxvf sendEmail-v1.56.tar.gz
cp sendEmail-v1.56/sendEmail /shell/check/
cat >> check.sh <span class="sh"><<EOF
#!/bin/bash
checkmail() {
/usr/bin/perl ./sendEmail -f userid@mail.com -t oneuserid@mail.com –cc twouserid@mail.com threeuserid@mail.com -u "$subject" -m "$data" -s smtp.mail.com -xu userid -xp password
sleep 300
}
while true;do
loadavg=`awk '{print $2}' /proc/loadavg`
diskuse=`df |awk '/cache/{print $5}'`
servrun=`netstat -pln|awk -F/ '/:80/{print $NF}'`
ip=`ifconfig|awk '/cast/{print $2}'|awk -F: '{if(NR==1){a=$2}else if(NR==2){b=$2}}END{print b"-"a}'`
data=`echo -e "ip:$ip\nloadavg/5min:$loadavg\tcacheuse%:$diskuse\tservice:$servrun"`
diskper=`echo $diskuse|sed 's/%//'`
num=`ps aux|grep check.sh|grep -v grep|wc -l`
if [[ $num > 2 ]];then
break 2
fi
if [[ $loadavg > 1.00 ]] &amp;&amp; [[ $diskper > 90 ]];then
subject="warning-$ip-loadavg-disk"
checkmail
else if [[ $loadavg > 1.00 ]];then
subject="warning-$ip-loadavg"
checkmail
else if [[ $diskper > 90 ]];then
subject="warning-$ip-disk"
checkmail
fi
sleep 60
done
EOF
</span></code></pre>
</div>
<p>完成,执行sh check.sh &> /dev/null即可。报警邮件如下:</p>
<p>标题:warning-192.168.0.100-10.10.10.10-disk<br />
ip:192.168.0.100-10.10.10.10<br />
loadavg/5min:0.38 cacheuse%:94% service:nginx</p>
nginx负载均衡(url_hash)配置
2010-03-02T00:00:00+08:00
nginx
http://chenlinux.com/2010/03/02/loadbalance-in-nginx-using-url_hash
<p>nginx是著名的非专职全七层负载均衡器,在用惯了四层LVS后,终于碰上了麻烦:LVS后端的4台RS磁盘都较小(20G),跑不到一天就塞满了东西;而根据预估,实际上一天时间该节点也就只有20G的文件增长。很显然,因为lvs转发的轮询算法,导致RS重复缓存了相同的文件。</p>
<p>针对这个情况,可以有两个办法(我只想到两个,欢迎大家补充):</p>
<ol>
<li>架构拆分,把不同的几个域名分别指向不同的server,这个在DNS上就能完成,不过就丧失了lvs的冗余;也可以用nginx的upstream+server配置,分别指向不通的RS,不过不同域名文件数量如果相差比较大的话,RS的负载就不均衡了……</li>
<li>url_hash,采用HAproxy的loadbalance uri或者nginx的upstream_hash模块,都可以做到针对url进行哈希算法式的负载均衡转发。</li>
</ol>
<p>那么,就开始试试nginx的url_hash负载均衡吧:</p>
<ol>
<li>安装部署:<br />
<code class="highlighter-rouge">bash
wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.01.tar.gz
tar zxvf pcre-8.01.tar.gz
wget http://wiki.nginx.org/images/7/78/Nginx_upstream_hash-0.3.tar.gz
tar zxvf Nginx_upstream_hash-0.3.tar.gz
wget http://sysoev.ru/nginx/nginx-0.7.65.tar.gz
tar zxvf nginx-0.7.65.tar.gz
</code><br />
vi nginx-0.7.65/src/http/ngx_http_upstream.h<br />
```c<br />
struct ngx_http_upstream_srv_conf_s {<br />
ngx_http_upstream_peer_t peer;<br />
void **srv_conf;</li>
</ol>
<p>ngx_array_t <em>servers; /</em> ngx_http_upstream_server_t */</p>
<p>+ngx_array_t *values;<br />
+ngx_array_t *lengths;<br />
+ngx_uint_t retries;</p>
<p>ngx_uint_t flags;<br />
ngx_str_t host;<br />
u_char *file_name;<br />
ngx_uint_t line;<br />
in_port_t port;<br />
in_port_t default_port;<br />
};<br />
```</p>
<p>为了安全,可以修改一下nginx的version信息:vi nginx-0.7.65/src/core/nginx.h</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="cp">#define NGINX_VERSION "2.6.STABLE21"
#define NGINX_VER "squid/" NGINX_VERSION
</span></code></pre>
</div>
<p>vi nginx-0.7.65/src/http/ngx_http_header_filter_module.c</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">static</span> <span class="kt">char</span> <span class="n">ngx_http_server_string</span><span class="p">[]</span> <span class="o">=</span> <span class="s">"Server: squid/2.6.STABLE21"</span> <span class="n">CRLF</span><span class="p">;</span>
</code></pre>
</div>
<p>vi nginx-0.7.65/src/http/ngx_http_special_response.c</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">static</span> <span class="n">u_char</span> <span class="n">ngx_http_error_tail</span><span class="p">[]</span> <span class="o">=</span>
<span class="s">"<hr><center>squid/2.6.STABLE21</center>"</span> <span class="n">CRLF</span>
<span class="s">"</body>"</span> <span class="n">CRLF</span>
<span class="s">"</html>"</span> <span class="n">CRLF</span>
</code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nb">cd </span>pcre-8.01
./configure --prefix<span class="o">=</span>/usr
make <span class="o">&&</span> make install
<span class="nb">cd </span>nginx-0.7.65
./configure --prefix<span class="o">=</span>/home/nginx --with-pcre --with-http_stub_status_module --with-http_ssl_module --without-http_rewrite_module --add-module<span class="o">=</span>/tmp/nginx_upstream_hash-0.3
</code></pre>
</div>
<p>vi auto/cc/gcc</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="cp"># debug
#CFLAGS="$CFLAGS -g"
</span></code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>make <span class="o">&&</span> make install
</code></pre>
</div>
<p>这样就安装完成了。</p>
<p>2、配置文件</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">upstream</span> <span class="s">images6.static.com</span> <span class="p">{</span>
<span class="kn">server</span> <span class="nf">11.11.11.11</span><span class="p">:</span><span class="mi">80</span><span class="p">;</span>
<span class="kn">server</span> <span class="nf">11.11.21.12</span><span class="p">:</span><span class="mi">80</span><span class="p">;</span>
<span class="kn">server</span> <span class="nf">11.11.21.13</span><span class="p">:</span><span class="mi">80</span><span class="p">;</span>
<span class="kn">server</span> <span class="nf">11.11.21.14</span><span class="p">:</span><span class="mi">80</span><span class="p">;</span>
<span class="kn">hash </span> <span class="nv">$request_uri</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">server</span> <span class="p">{</span>
<span class="kn">listen </span> <span class="mi">80</span><span class="p">;</span>
<span class="kn">server_name </span> <span class="s">images6.static.com</span><span class="p">;</span>
<span class="kn">access_log </span> <span class="n">/dev/null </span> <span class="s">main</span><span class="p">;</span>
<span class="kn">location</span> <span class="n">/</span> <span class="p">{</span>
<span class="kn">proxy_pass </span> <span class="s">http://images6.static.com</span><span class="p">;</span>
<span class="kn">proxy_set_header </span> <span class="s">Host </span> <span class="nv">$host</span><span class="p">;</span>
<span class="kn">proxy_set_header </span> <span class="s">X-Real-IP </span> <span class="nv">$remote_addr</span><span class="p">;</span>
<span class="kn">proxy_set_header </span> <span class="s">X-Forwarded-For</span> <span class="nv">$proxy_add_x_forwarded_for</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>以上配置的问题:</p>
<ol>
<li>RS中不能设置nginx本机的其他端口,我本来设定的server 11.11.11.10:3128,希望能把nginx本机也开上squid,省出一台机器来。结果在确认配置了DNS的情况下,返回状态码全是503……</li>
<li>RS一旦有宕机的,nginx不会重算hash,导致部分url返回错误信息;而启用hash_again标签的话,其他RS就都乱了。</li>
<li>RS中默认logformat中将显示nginx的IP。</li>
</ol>
<p>解决办法:<br />
1. 不知道<br />
2. 不采用hash_again标签而采用error_page重定向到专门的备份服务器保障访问<br />
3. 修改RS的logformat,把%>a改成%{X-Real_IP}>h即可。</p>
<p>最后的根本性问题:</p>
<p>对nginx下的RS集群进行增减操作,是否会对hash表产生影响?nginx_upstream_hash目录中的CHANGES有如下三条:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Changes with upstream_hash 0.3 06 Aug 2008
*) Bugfix: infinite loop when retrying after a 404 and the "not_found" flag of *_next_upstream was set.
*) Change: no more "hash_method" directive. Hash method is always CRC-32.
*) Change: failover strategy is compatible with PECL Memcache.
</code></pre>
</div>
<p>nginx的wiki上,关于hash_again的doc这么写到:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Number of times to rehash the value and choose a different server if the backend connection fails. Increase this number to provide high availability.
</code></pre>
</div>
<p>关于PECL Memcache,请参考下列链接:</p>
<p><a title="http://www.surfchen.org/archives/348" href="http://www.surfchen.org/archives/348">http://www.surfchen.org/archives/348</a><br />
<a href="http://tech.idv2.com/2008/07/24/memcached-004/">http://tech.idv2.com/2008/07/24/memcached-004/</a></p>
<p>尤其是第二个链接,其中关于rehash的解释,很好的解释了为什么大家都不推荐使用hash_again标签。<br />
由此可知,upstream_hash模块,使用的是余数计算standard+CRC32方式,10+2的存活率是17%,3+1的存活率是23%!<br />
而存活率最高的是consistent+CRC32方式,存活率是n/(n+m)*100%,10+2是83%,3+1是75%。</p>
<p>nginx的wiki中,还有另一个3rd模块upstream_consistent_hash,下回可以试试;<br />
网上还有针对upstream_hash模块的补丁<a href="http://www.sanotes.net/wp-content/uploads/2009/06/nginx_upstream_hash.pdf">http://www.sanotes.net/wp-content/uploads/2009/06/nginx_upstream_hash.pdf</a>,好模块就是有人研究呀~~</p>
wordpress部署时碰到的php小问题~
2010-02-25T00:00:00+08:00
web
php
http://chenlinux.com/2010/02/25/php-problem-of-wordpress-install
<p>看到windows live writer也支持wordpress,感觉这个博客系统确实流行,决定自己也试一把~install时,出了点问题,选择好sqlname和user、passwd信息后,另存为wp-config.php,next生成管理账户和密码的页面,居然顶上出现了“Warning: Cannot modify header information - headers already sent by …”,无视之,记下随机密码,下一步login——彻底废了,整个页面都是这个warning提示~~</p>
<p>百度了一下这个问题,原因真是多种多样,php不支持UTF8的BOM、php的output_buffering没打开、php中setcookie的使用限制等等,不过我这是从wp自己的页面文本框里复制出来的东东,应该问题不大才对。结果回去一看,原来是最最简单的问题:<?php...?>后面,多了一个空行!!</p>
<p>ctrl+a后ctrl+c复制文本,一般都会多出来一个\r\n,insert进vi的时候自然也就多了一个空行,不巧的是,在include或者require的php里,如果首尾有空行的话,程序就很有可能出问题…………</p>
<p>del掉这个空行,退出刷新页面,OK!</p>
memcached部署
2010-02-25T00:00:00+08:00
web
http://chenlinux.com/2010/02/25/memcached-install
<div class="highlighter-rouge"><pre class="highlight"><code>wget http://www.monkey.org/~provos/libevent-1.4.13-stable.tar.gz
wget http://memcached.googlecode.com/files/memcached-1.4.4.tar.gz
wget http://pecl.php.net/get/memcache-2.2.5.tgz
tar zxvf libevent-1.4.13-stable.tar.gz
tar zxvf memcached-1.4.4.tar.gz
<span class="nb">cd </span>libevent-1.4.13-stable
./configure --prefix<span class="o">=</span>/usr
make
make install
ln -s /usr/lib/libevent-1.4.so.2 /usr/lib64/libevent-1.4.so.2
<span class="nb">cd </span>memcached-1.4.4
./configure --prefix<span class="o">=</span>/home/memcached --enable-64bit
make
make install
/home/memcached/bin/memcached -d -m 1024 -p 11211 -u root
</code></pre>
</div>
<p>参数说明:<br />
-d 启动为守护进程<br />
-m <num> 分配给Memcached使用的内存数量,单位是MB,默认为64MB
-u <username> 运行Memcached的用户,仅当作为root运行时
-l <ip_addr> 监听的服务器IP地址,默认为环境变量INDRR_ANY的值
-p <num> 设置Memcached监听的端口,最好是1024以上的端口
-c <num> 设置最大并发连接数,默认为1024
-P <file> 设置保存Memcached的pid文件,与-d选择同时使用</file></num></num></ip_addr></username></num></p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nb">cd </span>memcache-2.2.5
/home/php/bin/phpize
./configure --prefix<span class="o">=</span>/home/phpmemcache --with-php-config<span class="o">=</span>/home/php/bin/php-config
make
make install
sed –i ‘s:./:/home/php/lib/php/extensions/no-debug-zts-20060613/:g’ /home/php/lib/php.ini
sed –i ‘/zip.dll/aextension<span class="o">=</span>memcache.so’ /home/php/lib/php.ini
cat >> /cache/data/test.php <span class="sh"><<EOF
<?php
$memcache = new Memcache;
/*$memcache->connect('localhost', 11211) or die ("Could not connect");
/*addServer方式更能体现分布式的优势,也完成故障转移,就是转移的时候HIT要重新开始*/
$memcache->addServer('127.0.0.1', 11211);
$memcache->addServer('192.168.0.2', 11212);
$version = $memcache->getVersion();
echo "Server's version: ".$version."<br/>n";
$tmp_object = new stdClass;
$tmp_object->str_attr = 'test';
$tmp_object->int_attr = 123;
$memcache->set('key', $tmp_object, false, 10) or die ("Failed to save data at the server");
echo "Store data in the cache (data will expire in 10 seconds)<br/>n";
$get_result = $memcache->get('key');
echo "Data from the cache:<br/>n";
var_dump($get_result);
?>
EOF
</span></code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>curl <a <span class="nv">href</span><span class="o">=</span><span class="s2">"http://localhost/test.php"</span>>http://localhost/test.ph</a>
Server<span class="s1">'s version: 1.4.4<br/>
Store data in the cache (data will expire in 10 seconds)<br/>
Data from the cache:<br/>
object(stdClass)#3 (2) {
["str_attr"]=>
string(4) "test"
["int_attr"]=>
int(123)
}
# telnet localhost 11211
</span></code></pre>
</div>
<div class="highlighter-rouge"><pre class="highlight"><code>Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
stats
STAT pid 3015
STAT uptime 3185 memcached运行的秒数
STAT time 1267090234
STAT version 1.4.4
STAT pointer_size 64
STAT rusage_user 0.000000
STAT rusage_system 0.000000
STAT curr_connections 10
STAT total_connections 18
STAT connection_structures 11
STAT cmd_get 3 查询缓存的次数
STAT cmd_set 5 设置key=>value的次数
STAT cmd_flush 0
STAT get_hits 3 缓存命中的次数
STAT get_misses 0
STAT delete_misses 0
STAT delete_hits 0
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read 2697
STAT bytes_written 1150
STAT limit_maxbytes 1073741824
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT threads 4
STAT conn_yields 0
STAT bytes 1255
STAT curr_items 2
STAT total_items 5
STAT evictions 0
END
</code></pre>
</div>
<p>完成~~~以上仅为试验,如果只是一个小web单机环境的话,只需要php+eaccelerator(或者apc/xcache等)就足够了。用php+memcached+mysql-proxy+mysql是大型网站架构的事情~~</p>
cacti安装记录
2010-02-24T00:00:00+08:00
monitor
cacti
http://chenlinux.com/2010/02/24/cacti-install
<p>cacti运行在lamp环境下,采用net-snmp获得监控数据,由rrdtool绘图。所以cacti的安装,主要就是apache、mysql、php、rrdtool、net-snmp这几个的安装。其中apache2是我们服务器上早就有的,可以利用;而原有的php5因为不支持mysql,所以要重新编译。</p>
<ol>
<li>mysql安装:</li>
</ol>
<div class="highlighter-rouge"><pre class="highlight"><code>wget http://mysql.cs.pu.edu.tw/Downloads/MySQL-5.1/mysql-5.1.44.tar.gz
tar zxvf mysql-5.1.44.tar.gz
<span class="nb">cd </span>mysql-5.1.44
groupadd mysql
useradd mysql -g mysql
./configure --prefix<span class="o">=</span>/home/mysql --with-unix-socket-path<span class="o">=</span>/tmp/mysql.sock --localstatedir<span class="o">=</span>/cache/mysql --enable-assembler --with-mysqld-ldflags<span class="o">=</span>-all-static --with-client-ldflags<span class="o">=</span>-all-static --with-extra-charsets<span class="o">=</span>gbk,gb2312,utf8 --enable-thread-safe-client --with-big-tables --enable-local-infile --with-ssl --with-mysqld-user<span class="o">=</span>mysql
make
make install
<span class="nb">cd </span>support-files/
cp my-medium.cnf /etc/my.cnf
cp mysql.server /etc/rc.d/init.d/mysqld
<span class="nb">cd</span> ../scripts/
./mysql_install_db --user<span class="o">=</span>mysql
mkdir –p /cache/mysql
chown -R mysql.mysql /cache/mysql/
chgrp -R mysql /home/mysql/
chmod 700 /etc/rc.d/init.d/mysqld
ln -s /etc/rc.d/init.d/mysqld /etc/rc.d/rc3.d/S97mysqld
chmod 777 /tmp/
/home/mysql/bin/mysqld_safe --user<span class="o">=</span>mysql &amp;
ln –s /home/mysql/bin/<span class="k">*</span> /usr/bin/
sed -i /^myisam/aset-variable<span class="o">=</span><span class="nv">wait_timeout</span><span class="o">=</span>200 /etc/my.cnf
sed -i /^myisam/aset-variable<span class="o">=</span><span class="nv">max_user_connections</span><span class="o">=</span>500 /etc/my.cnf
sed -i /^myisam/aset-variable<span class="o">=</span><span class="nv">max_connections</span><span class="o">=</span>1000 /etc/my.cnf
/etc/init.d/mysqld restart
</code></pre>
</div>
<ol>
<li>php安装</li>
</ol>
<div class="highlighter-rouge"><pre class="highlight"><code>wget http://cn.php.net/distributions/php-5.2.12.tar.gz
tar zxvf php-5.2.12.tar.gz
<span class="nb">cd </span>php-5.2.12
./configure --prefix<span class="o">=</span>/home/php --with-apxs2<span class="o">=</span>/home/apache2/bin/apxs --with-mysql<span class="o">=</span>/home/mysql --enable-sockets --with-zlib-dir<span class="o">=</span>/usr/include --with-gd --with-snmp --enable-ucd-snmp-hack --with-ttf --enable-mbstring --enable-xml --with-mysql-sock<span class="o">=</span>/tmp/mysql.sock
<span class="c"># (注:apache版本不同,--with-apxs2可能要写成--with-apxs)</span>
make
make install
cp php.ini-dist /home/php/lib/php.ini
ln -s /home/php/bin/<span class="k">*</span> /usr/local/bin/
</code></pre>
</div>
<p>3、apache检测</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># grep php /home/apache2/conf/httpd.conf</span>
DirectoryIndex index.php index.html index.htm
AddType application/x-httpd-php .php
LoadModule php5_module modules/libphp5.so
<span class="c"># /home/apache2/bin/apachectl configtest</span>
Syntax OK
<span class="c"># cat >> /cache/data/index.php <<EOF</span>
<?php
phpinfo<span class="o">()</span>;
?>
EOF
<span class="c"># curl http://localhost | grep module_mysql</span>
<h2><a <span class="nv">name</span><span class="o">=</span><span class="s2">"module_mysql"</span>>mysql</a></h2>
</code></pre>
</div>
<p>4、rrdtool安装<br />
麻烦东西来了,网上很多cacti部署教程,都在rrdtool上大费周章,因为这个东东依赖的库文件很多,而且自己本身的版本不同,库文件的种类和版本要求也不一样。首先,尽可能的把这些东西都安装吧:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>rpm -qa|grep lm_sensors
rpm -qa|grep beecrypt
rpm -qa|grep libpng
rpm -qa|grep elfutils
rpm -qa|grep sensors
rpm -qa|grep pixman
rpm -qa|grep freetype
rpm -qa|grep fontconfig
rpm -qa|grep net-snmp
rpm -qa|grep libart_lgpl
rpm -qa|grep zlib
rpm -qa|grep glib
rpm -qa|grep libxml2
rpm -qa|grep intltool
rpm -qa|grep cairo
rpm -qa|grep pango
</code></pre>
</div>
<h1 id="find--name-pangocairopccairopangocairopango">find / –name pangocairo.pc,如果没有,就要把cairo和pango重装了,务必先cairo后pango。</h1>
<p>如果以上齐全,可以去http://oss.oetiker.ch/rrdtool/pub/libs下载rrdtool的源码编译,然后按照make的warning信息慢慢调整库文件的相应版本号去了……</p>
<p>如果不要求自己成为编译达人,只求搞定的,那么按照如下办法,轻松搞定吧:<br />
```bash<br />
# cat > /etc/yum.repos.d/ct5_64.repo «EOF<br />
[base]<br />
name=CentOS-5.4 - Base<br />
baseurl=http://mirrors.163.com/centos/5.4/os/x86_64/<br />
gpgcheck=1<br />
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5<br />
EOF<br />
# yum install ruby xorg-x11-fonts-Type1</p>
<h1 id="cat--etcyumreposdct564repo-eof">cat > /etc/yum.repos.d/ct5_64.repo «EOF</h1>
<p>[base]<br />
name=CentOS-5.4 – Base<br />
baseurl=http://apt.sw.be/redhat/el$releasever/en/$basearch/dag<br />
gpgcheck=1<br />
gpgkey=http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt<br />
enabled=1<br />
EOF<br />
# rpm –import <a href="http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt">http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt</a><br />
# yum install rrdtool<br />
```<br />
(还嫌不够简单?那还有更简单的:<br />
wget http://dag.wieers.com/rpm/packages/rpmforge-release/rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm;rpm -Uvh rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm;yum install rrdtool rrdtool-php即可)</p>
<p>5、cacti安装<br />
<code class="highlighter-rouge">bash
# wget http://www.cacti.net/downloads/cacti-0.8.7e.tar.gz
# tar zxvf cacti-0.8.7e.tar.gz –C /cache/data/
# mv /cache/data/cacti-0.8.7e /cache/data/cacti
# cd /cache/data/cacti
# mysql -uroot –p
mysql> create database cacti;
mysql> grant all privileges on cacti.* to <a href="mailto:cacti@&quot;localhost">cacti@"localhost</a>" identified by '123456';
mysql> flush privileges;
# /home/mysql/bin/mysql -ucacti -p cacti < cacti.sql
# useradd cacti -d /cache/data/cacti -s /bin/false
# chown -R cacti rra
# chown -R cacti log
# sed –i 's/username = "cactiuser/username = "cacti/' include/config.php
# sed –i 's/password = "cactiuser/password = "123456/' include/config.php
# echo '*/5 * * * * /home/php/bin/php /cache/data/cacti/poller.php &amp;> /dev/null' >>/var/spool/cron/root
</code></p>
<p>6、web页面发布<br />
在httpd.conf中发布<br />
<code class="highlighter-rouge">apache
<Directory /cache/data>
Options Indexes FollowSymLinks
AllowOverride None
Order Allow,Deny
Allow from all
</Directory>
</code></p>
<p>大功告成,接下来都是鼠标的事了,在browser中登陆http://yourdomian/cacti,按实际情况修改php/mysql/net-snmp的which和version信息,一路next即可,最后,cacti的初始用户名密码都是admin。</p>
shell变量扩展
2010-02-23T00:00:00+08:00
bash
http://chenlinux.com/2010/02/23/extend-shell-variables
<p>第一种扩展形式,按长度截取:${PARAMETER:OFFSET:LENGTH};例:</p>
<p>i=http://www.baidu.com/a/b.html;j=${i:1:10};echo $j<br />
ttp://www.</p>
<p>第二种扩展形式,按模式截取:${PARAMETER#WORD}、${PARAMETER##WORD}、${PARAMETER%WORD}、${PARAMETER%%WORD};例:</p>
<p>i=http://www.baidu.com/a/b.html;b=${i%/<em>};a=${i%%/</em>};c=${i#<em>/};d=${i##</em>/};echo $b;echo $c;echo $d;echo $a<br />
http://www.baidu.com/a<br />
/www.baidu.com/a/b.html<br />
b.html<br />
http:</p>
<p>第三种扩展形式,按模式替换:${PARAMETER/PATTERN/STRING};${PARAMETER//PATTERN/STRING},例:</p>
<p>i=http://www.baidu.com/a/b.html;x=${i/baidu/google};y=${i//?a/xyz};echo $x;echo $y<br />
http://www.google.com/a/b.html<br />
http://www.xyzidu.comxyz/b.html</p>
<p>第四种扩展形式,指定默认值:${PARAMETER:-WORD}、${PARAMETER:+WORD}、${PARAMETER:?WORD}、${PARAMETER:=WORD},例:</p>
<p>unset x;y=”abc def”; echo “/${x:-‘XYZ’}/${y:-‘XYZ’}/$x/$y/”<br />
/’XYZ’/abc def//abc def/</p>
<p>unset x;y=”abc def”; echo “/${x:=’XYZ’}/${y:=’XYZ’}/$x/$y/”<br />
/’XYZ’/abc def/’XYZ’/abc def/</p>
<p>( unset x;y=”abc def”; echo “/${x:?’XYZ’}/${y:?’XYZ’}/$x/$y/” ) >so.txt 2>se.txt<br />
cat so.txt<br />
cat se.txt<br />
-bash: x: XYZ</p>
<p>unset x;y=”abc def”; echo “/${x:+’XYZ’}/${y:+’XYZ’}/$x/$y/”<br />
//’XYZ’//abc def/</p>
<p>说明:-返回默认值,但不更改变量本身;=返回默认值同时更改变量为默认值;?返回默认值到标准错误;+与-相反。</p>
ESI 语言介绍
2010-02-22T00:00:00+08:00
CDN
squid
http://chenlinux.com/2010/02/22/esi-cache
<p>今天看到一个叫做 ESI 的东东,以此为线索,一路链接下去,颇为有趣,摘抄些新闻/博客段落来,算作长眼了:</p>
<p>首先是《CSDN09大会见闻》,其中提到康神的讲演《网站的那些事儿》,介绍了大规模网站架构上的种种工具和技术“比如做数据切分的 Mysql Proxy,分布式缓存的 MemoryCached,Web 服务器缓存的 Squid,页面优化的 Y!Slow,页面切分方面的 js 拼接和 iframe 拼接(呵呵,怪不得搜狐的页面都是 js 加载,对搜索引擎很不友好),前端服务器 Lighttpd, Squid3, ESI 等。也提到了架构设计中扩展性,可用性和一致性三者的关系,优化时应让让一致性延迟。提到目前网站系统一般都是三层结构 DB 层,逻辑层和前端层,而可扩展度方面DB < 逻辑 < 前端,原因就是有状态的最难扩展,无状态的最容易扩展,所以主张优化时尽量减少DB中存储的状态,将逻辑前移,最后总结优化的大方向就是:逻辑前移,善用缓存(无处不在的缓存)和数据冗余(方便查询)。”</p>
<p>从中得到几个信息——</p>
<ol>
<li>搜狐的架构康神插手了;</li>
<li>squid对ESI的支持限于3.0版本以上(这点很重要,因为其他任何地方都没写明);</li>
<li>缓存无处不在~~</li>
</ol>
<p>然后是ESI的概念原理。</p>
<p><strong>“ESI(Edge Side Include)通过使用简单的标记语言来对那些可以加速和不能加速的网页中的内容片断进行描述,每个网页都被划分成不同的小部分分别赋予不同的缓存控制策略,使Cache服务器可以根据这些策略在将完整的网页发送给用户之前将不同的小部分动态地组合在一起。通过这种控制,可以有效地减少从服务器抓取整个页面的次数,而只用从原服务器中提取少量的不能缓存的片断,因此可以有效降低原服务器的负载,同时提高用户访问的响应时间”</strong></p>
<p>从中得到几个信息——</p>
<ol>
<li>ESI不是一门新语言,而只是嵌入在html里的一种标记,首先要求对网页进行模板式的规划,然后由支持ESI的cache服务器根据“HTTP请求标题或用户的cookie信息”自行组装返回给browser;</li>
<li>我想到lighttpd和nginx的流媒体支持,也是根据cookie信息,获取拖拽的指定帧定位。不过没找到这两个对esi的cache处理文章,毕竟他们的主业还是webserver……</li>
</ol>
<p>然后是计算机世界网中关于ESI的介绍。内容较长,不贴了,要点如下:</p>
<ol>
<li>ESI最多支持三级递归,可以在包含文档中再嵌套进一步ESI标记;</li>
<li>ESI支持基于布尔比较或者环境变量的条件处理;</li>
<li>ESI支持cgi环境变量,比如cookie;</li>
<li>ESI支持开发者定制失败动作(这不就是我梦寐以求的东东?呵呵,夸张鸟~)</li>
<li>ESI提供内容无效规范进行内容管理(cache的purge不再头疼?);</li>
<li>ESI是基于XML语言的,现有一个java的定制标签库JESI帮助生成jsp代码,这个东东我想是ESI标准被诸多寡头接受的重要原因,不然去哪找来一大批专门写ESI的设计员呢;</li>
<li>ESI的主要开发和推动者,是CDN老大akaimai和DB老大oracle!!再次验证康神的话,Optimization=Cache+Data。</li>
</ol>
<p>最后,说一下squid3如何支持ESI。</p>
<p>./configure参数如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>--enable-esi
Enable ESI for accelerators. Requires libexpat.
Enabling ESI will cause squid to follow the Edge
Acceleration Specification (www.esi.org). This
causes squid to IGNORE client Cache-Control headers.
DO NOT use this in a squid configured as a web
proxy, ONLY use it in a squid configured for
webserver acceleration.
</code></pre>
</div>
<p>squid.conf参数如下:</p>
<pre><code class="language-squid"> httpd_accel_surrogate_id unset-id
http_accel_surrogate_remote on
esi_parser custom
</code></pre>
<p>写到最后郁闷了一下,这个东东,为了cache而发明出来了,却是得写在web上的。那我这篇博文,该归哪个类别呢?——或者这也验证了工作中的一点郁闷吧,cache的问题,经常出在web上,我们做CDN的,能怎么办呢?</p>
<p>ESI网站:</p>
<p><a href="http://www.akamai.com/html/support/esi.html">http://www.akamai.com/html/support/esi.html</a><br />
<a href="http://www.w3.org/TR/esi-lang">http://www.w3.org/TR/esi-lang</a></p>
<p><strong>2010年8月1日:</strong></p>
<h1 id="esi">ESI指令集</h1>
<h3 id="include-">include 标签</h3>
<p>首先尝试 include 页面 <code class="highlighter-rouge">1.html</code>,如果不存在就显示 <code class="highlighter-rouge">2.html</code>,如果还不存在,就忽略这条 esi</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nt"><esi:include</span> <span class="na">src=</span><span class="s">"http://example.com/1.html"</span> <span class="na">alt=</span><span class="s">"http://bak.example.com/2.html"</span> <span class="na">onerror=</span><span class="s">"continue"</span><span class="nt">/></span>
</code></pre>
</div>
<h3 id="case-when-">case when 标签</h3>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nt"><esi:choose></span>
<span class="nt"><esi:when</span> <span class="na">test=</span><span class="s">"$(HTTP_COOKIE{group})=='Advanced'"</span><span class="nt">></span>
<span class="nt"><esi:include</span> <span class="na">src=</span><span class="s">"http://www.example.com/advanced.html"</span><span class="nt">/></span>
<span class="nt"></esi:when></span>
<span class="nt"><esi:when</span> <span class="na">test=</span><span class="s">"$(HTTP_COOKIE{group})=='Basic User'"</span><span class="nt">></span>
<span class="nt"><esi:include</span> <span class="na">src=</span><span class="s">"http://www.example.com/basic.html"</span><span class="nt">/></span>
<span class="nt"></esi:when></span>
<span class="nt"><esi:otherwise></span>
<span class="nt"><esi:include</span> <span class="na">src=</span><span class="s">"http://www.example.com/new_user.html"</span><span class="nt">/></span>
<span class="nt"></esi:otherwise></span>
<span class="nt"></esi:choose></span>
</code></pre>
</div>
<h3 id="try-catch-">try catch 标签</h3>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nt"><esi:try></span>
<span class="nt"><esi:attempt></span>
<span class="nt"><esi:comment</span> <span class="na">text=</span><span class="s">"Include an ad"</span><span class="nt">/></span>
<span class="nt"><esi:include</span> <span class="na">src=</span><span class="s">"http://www.example.com/ad1.html"</span><span class="nt">/></span>
<span class="nt"></esi:attempt></span>
<span class="nt"><esi:except></span>
<span class="nt"><esi:comment</span> <span class="na">text=</span><span class="s">"Just write some HTML instead"</span><span class="nt">/></span>
<span class="nt"><a</span> <span class="na">href=</span><span class="s">www.akamai.com</span><span class="nt">></span>www.example.com<span class="nt"></a></span>
<span class="nt"></esi:except></span>
<span class="nt"></esi:try></span>
</code></pre>
</div>
<h3 id="remove-">remove 标签</h3>
<p>如果服务器可执行 ESI,就只执行 include;如果不可,只好识别标准的HTML</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nt"><esi:include</span> <span class="na">src=</span><span class="s">"http://www.example.com/ad.html"</span><span class="nt">/></span>
<span class="nt"><esi:remove></span>
<span class="nt"><a</span> <span class="na">href=</span><span class="s">"http://www.example.com"</span><span class="nt">></span>www.example.com<span class="nt"></a></span>
<span class="nt"></esi:remove></span>
</code></pre>
</div>
<h3 id="esi-1">esi变量</h3>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="nt"><esi:vars></span>
<span class="nt"><img</span> <span class="na">src=</span><span class="s">"http://www.example.com/$(HTTP_COOKIE{type})/hello.gif"</span> <span class="nt">/></span>
<span class="nt"></esi:vars></span>
</code></pre>
</div>
<p>变量都是从request-header中获得的,squid只支持标准的esi协议,即只识别下列header。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="err"> </span><span class="n">addVariable</span> <span class="p">(</span><span class="s">"HTTP_ACCEPT_LANGUAGE"</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="n">new</span> <span class="n">ESIVariableLanguage</span><span class="p">);</span>
<span class="err"> </span><span class="n">addVariable</span> <span class="p">(</span><span class="s">"HTTP_COOKIE"</span><span class="p">,</span> <span class="mi">11</span><span class="p">,</span> <span class="n">new</span> <span class="n">ESIVariableCookie</span><span class="p">);</span>
<span class="err"> </span><span class="n">addVariable</span> <span class="p">(</span><span class="s">"HTTP_HOST"</span><span class="p">,</span> <span class="mi">9</span><span class="p">,</span> <span class="n">new</span> <span class="n">ESIVariableHost</span><span class="p">);</span>
<span class="err"> </span><span class="n">addVariable</span> <span class="p">(</span><span class="s">"HTTP_REFERER"</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="n">new</span> <span class="n">ESIVariableReferer</span><span class="p">);</span>
<span class="err"> </span><span class="n">addVariable</span> <span class="p">(</span><span class="s">"HTTP_USER_AGENT"</span><span class="p">,</span> <span class="mi">15</span><span class="p">,</span> <span class="n">new</span> <span class="n">ESIVariableUserAgent</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">));</span>
<span class="err"> </span><span class="n">addVariable</span> <span class="p">(</span><span class="s">"QUERY_STRING"</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="n">new</span> <span class="n">ESIVariableQuery</span><span class="p">(</span><span class="n">uri</span><span class="p">));</span>
</code></pre>
</div>
<h3 id="section">逻辑表达式</h3>
<p><code class="highlighter-rouge">==</code>,<code class="highlighter-rouge">!=</code>,<code class="highlighter-rouge"><</code>,<code class="highlighter-rouge">></code>,<code class="highlighter-rouge"><=</code>,<code class="highlighter-rouge">>=</code>,<code class="highlighter-rouge">!</code>,<code class="highlighter-rouge">&</code>,<code class="highlighter-rouge">|</code>都只能用于表达式之间的关系,本身不具有真义,所以类似shell的(1 & 2)是不成立的。</p>
<p>ESI必须由web端在response-header中声明Surrogate-Control: content=”ESI/1.0”。启用ESI后,Set-Cookie、Cache-Control、Last-Modified无效。</p>
qhttpd
2010-02-09T00:00:00+08:00
web
http://chenlinux.com/2010/02/09/qhttpd
<p>今天偷菜极不顺利,QQ空间老坏,看着那个Qhttpd的502提示页面就郁闷,决定自己也安一个看看。<br />
<a href="http://www.qdecoder.org/qhttpd/download.qsp">http://www.qdecoder.org/qhttpd/download.qsp</a><br />
话说网上人都说QQ的视频服务器用的是qhttpd,Qzone用的是自己的qzhttpd呀。为啥农场的错误是qhttpd呢?<br />
以上,过完年在说。呵呵<br />
过年回来鸟~~去韩国下载了qhttpd来看,默认安装配置什么的,真的超级简单——功能也超级简单(连logformat都没有)。如果想要玩出花样来,好吧,从源代码开始,慢慢写C去吧~~~</p>
cache驻留时间(七、Vary)
2010-02-09T00:00:00+08:00
CDN
squid
http://chenlinux.com/2010/02/09/intro-vary
<p>之前的系列文章,只涉及squid本身,今天突然想到,其实这个网站加速,除了squid缓存这种缩短传输距离的办法以外,还有另一个办法——压缩内容以缩短传输时间。<br />
而糟糕的问题是,squid对压缩内容的缓存,限制多多。因为squid默认是以HTTP/1.0进行内容传输的,对HTTP/1.1协议兼容性不怎么滴~~一个不小心,browser就会接受到squid交给它的无法理解的内容,并忠实的把这个错乱信息显示给网民……然后你就等着客户投诉电话吧~~<br />
这个具体的限制就是:squid只支持静态压缩,不支持动态压缩。反应到header上,就是rep_header里必须指明Content-length是多少多少,不能采用Transfer-Encoding: chunked这样的动态块格式;rep_header里必须指明Vary是Accept-Encoding,而不能是其他的User-Agent等等。</p>
<p>HTTP/1.1标准中,是建议所有的网页都加上vary头的。可见这个东东的重要性。</p>
<p>我不知道是不是有其他的缓存服务器能够支持vary值为user-agent,不过却在网上看到这么一句话:“如果依照除请求头以外的其他条件决定是否使用压缩(例如:HTTP版本),你必须设置Vary头的值为”*“来完全阻止代理服务器的缓存”——我正好在今天看到一个客户的网站所有内容都加上了Vary: *,于是全部回源。。。</p>
<p>接下来,当源站把Content-Length和Accept-Encoding都设定好了,squid就万事大吉了么?嗯,理论上是的,不过最好还是检查一下配置文件,万一你前任或者同事在里头写了句“cache_vary off”,上面那些可就全做了无用功了……</p>
<p>检查完成,但看看分析结果,怎么缓存命中率还是不算高,访问速度还是不算快?(呃,你想要多高多快?)那,我这还有一招——隆重推出windtear的文章《[squid patch] 解决 Accept-Encoding不一致造成的多份缓存问题》,因为IE和FF在发出这个header的时候,其实内容是不一样的:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>IE : Accept-Encoding: gzip, deflate
FF : Accept-Encoding: gzip,deflate
</code></pre>
</div>
<p>注意到了么?IE的内容里多了一个空格!<br />
感谢windtear这些已经深入squid源代码的大神们,把src/http.c相关部分修改如下,编译完成即可:<br />
<code class="highlighter-rouge">c
strListAdd(&vstr, name, ',');
hdr = httpHeaderGetByName(&request->header, name);
value = strBuf(hdr);
if (value) {
+ if (strcmp(name, "accept-encoding") != 0) {
value = rfc1738_escape_part(value);
stringAppend(&vstr, "="", 2);
stringAppend(&vstr, value, strlen(value));
stringAppend(&vstr, """, 1);
+ } else {
+ if(strstr(value,"gzip") != NULL || strstr(value,"deflate") != NULL) {
+ stringAppend(&vstr, "="gzip,%20deflate"", 18);
+ }
+ }
}
+ safe_free(name);
stringClean(&hdr);
}
safe_free(request->vary_hdr);
</code></p>
squid运行分析
2010-02-06T00:00:00+08:00
squid
http://chenlinux.com/2010/02/06/squid-running-analysis
<p>上回编译加载tcmalloc后,效果各有不同,所以还得细分具体运行情况,以便之后继续优化。<br />
之前的架构是1个lvs下挂6台leaf+1台parent。现在已经给7台squid都加载tcmalloc了。leaf运行上佳,CPU占用率甚至降到了2%,loadavg也不过0.2。但parent的CPU占用率虽然降低,loadavg却依然在1以上——这对于单核服务器来说,可不是什么好事。分析日志,或者用squidclient分析cache情况,leaf如下:<br />
cat access.log |awk ‘{if(/NONE/){a[NONE]++}}END{print a[NONE]/NR}’<br />
0.981347<br />
squidclient -p 80 mgr:info<br />
Cache information for squid:<br />
Request Hit Ratios: 5min: 97.8%, 60min: 98.3%<br />
Byte Hit Ratios: 5min: 97.8%, 60min: 98.2%<br />
Request Memory Hit Ratios: 5min: 85.8%, 60min: 86.8%<br />
Request Disk Hit Ratios: 5min: 9.8%, 60min: 9.1%<br />
Storage Swap size: 19891740 KB<br />
Storage Mem size: 1048572 KB<br />
Mean Object Size: 9.67 KB<br />
可以看到缓存文件的平均大小不足10KB,绝大多数的请求都在内存中处理掉了。所以在加载了优化内存反应速度的tcmalloc后,效果那么明显。<br />
parent如下:<br />
$ cat access.log |awk ‘{if(/NONE/){a[NONE]++}}END{print a[NONE]/NR}’<br />
0.179209<br />
$ squidclient -p 80 mgr:info<br />
Cache information for squid:<br />
Request Hit Ratios: 5min: 31.1%, 60min: 32.3%<br />
Byte Hit Ratios: 5min: 38.4%, 60min: 36.9%<br />
Request Memory Hit Ratios: 5min: 7.8%, 60min: 12.2%<br />
Request Disk Hit Ratios: 5min: 32.7%, 60min: 37.9%<br />
Storage Swap size: 40300232 KB<br />
Storage Mem size: 524284 KB<br />
Mean Object Size: 11.68 KB<br />
只有30%的缓存命中,而且基本还都是从磁盘读取的(awk结果排除了REFRESH_HIT,所以更低)。难怪上次优化没什么效用了……<br />
为了保证服务,先给这组服务器加上了round-robin的双parent。新parent的硬件情况和老的一样。而squid配置上,则采用了aufs方式,不再使用diskd方式。运行到现在30个小时,分析如下:<br />
$ cat /cache/logs/access.log |awk ‘{if(/NONE/){a[NONE]++}}END{print a[NONE]/NR}’<br />
0.238754<br />
$ squidclient -p 80 mgr:info<br />
Cache information for squid:<br />
Request Hit Ratios: 5min: 22.7%, 60min: 22.8%<br />
Byte Hit Ratios: 5min: 22.9%, 60min: 20.1%<br />
Request Memory Hit Ratios: 5min: 22.2%, 60min: 24.3%<br />
Request Disk Hit Ratios: 5min: 64.4%, 60min: 65.0%<br />
Storage Swap size: 4640308 KB<br />
Storage Mem size: 1048588 KB<br />
Mean Object Size: 9.08 KB</p>
<p>看起来差不多的样子。<br />
因为确认mem没怎么用上,下一步看disk的I/O。<br />
采用diskd的parent如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[root@tinysquid2 ~]# iostat -x /dev/xvdb2 5 5
Linux 2.6.18-128.el5xen (tinysquid2)
02/06/2010
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.00 10.00 0.00 90.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
xvdb2
0.00 5.00 8.60 46.60 81.60 412.80 8.96 0.32 5.80 1.75 9.68
</code></pre>
</div>
<p>采用aufs的parent如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[root@tinysquid3 ~]# iostat -x /dev/xvdb2 5 5
Linux 2.6.18-128.el5xen (tinysquid3)
02/06/2010
avg-cpu: %user %nice %system %iowait %steal %idle
0.20 0.00 0.40 1.60 0.20 97.60
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
xvdb2
0.00 8.58 3.19 6.19 25.55 118.16 15.32 0.02 2.47 1.70 1.60
</code></pre>
</div>
<p>以上结果的解释如下:</p>
<p>rrqm/s: 每秒进行 merge 的读操作数目。即 delta(rmerge)/s <br />
wrqm/s: 每秒进行 merge 的写操作数目。即 delta(wmerge)/s <br />
r/s: 每秒完成的读 I/O 设备次数。即 delta(rio)/s <br />
w/s: 每秒完成的写 I/O 设备次数。即 delta(wio)/s <br />
rsec/s: 每秒读扇区数。即 delta(rsect)/s <br />
wsec/s: 每秒写扇区数。即 delta(wsect)/s <br />
rkB/s: 每秒读K字节数。是 rsect/s 的一半,因为每扇区大小为512字节。(需要计算) <br />
wkB/s: 每秒写K字节数。是 wsect/s 的一半。(需要计算) <br />
avgrq-sz: 平均每次设备I/O操作的数据大小(扇区)。delta(rsect+wsect)/delta(rio+wio) <br />
avgqu-sz: 平均I/O队列长度。即 delta(aveq)/s/1000(因为aveq的单位为毫秒)。 <br />
await: 平均每次设备I/O操作的等待时间 (毫秒)。即delta(ruse+wuse)/delta(rio+wio) <br />
svctm: 平均每次设备I/O操作的服务时间 (毫秒)。即 delta(use)/delta(rio+wio) <br />
%util: 一秒中有百分之多少的时间用于 I/O 操作,或者说一秒中有多少时间 I/O 队列是非空的。即 delta(use)/s/1000(因为use的单位为毫秒)</p>
<p>从上面的运行情况看,都是w操作为主,但diskd比aufs每秒w的次数要大,而每次w的服务时间也大——大的同时波动性也不太稳定——由此导致rw时的等待时间也延长——进一步的结果就是I/O非空时间变少——最后的结果就是disk的I/O压力变大!<br />
因为现在已经双parent,loadavg降低,所以不好看出之前的高loadavg问题关键。不过至少从现在的运行来看,aufs比diskd要好。</p>
<p>————————————————————————————————————————<br />
3月25日补充:</p>
<p>aufs虽然是异步io,但某些文件默认的写操作并不如此。需要在编译时修改src/fs/aufs/store_asyncufs.h中的#define ASYNC_WRIT值为1。<br />
对于aufs,可以使用squidclient mgr:squidaio_counts查看,其中queue一项,据说不应该超过线程数量的5倍。而线程数量跟cache_dir数量的关系如下:</p>
<p>cache_dirs Threads<br />
1 16<br />
2 26<br />
3 32<br />
4 36<br />
5 40<br />
6 44</p>
netfilter的conntrack优化
2010-02-06T00:00:00+08:00
linux
iptables
http://chenlinux.com/2010/02/06/conntrack-of-netfilter
<p><a href="http://www.wallfire.org/misc/netfilter_conntrack_perf.txt">参考原文</a><br />
<a href="http://www.linuxmine.com/5791.html">中文版</a><br />
说是zz,好歹也是自己看完以后的zz,所以组织语言次序都是自己来:<br />
首先解释两个概念性的名词</p>
<p>conntrack最大数量.叫做conntrack_max<br />
存储这些conntrack的hash表的大小,叫做hashsize<br />
hash表存在于固定的的不可swap的内存中.<br />
conntrack_mark决定占用多少这些不可swap的内存<br />
hashsize=conntrack_max/8=ramsize(in bytes)/131072/(x/32)<br />
x表示使用的指针类型是(32位还是64的)</p>
<p>读取conntrack_max值<br />
cat /proc/sys/net/ipv4/ip_conntrack_max</p>
<p>读取hashsize值<br />
cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets</p>
<p>ip_conntrack buffer使用情况<br />
grep conn /proc/slabinfo</p>
<p>文章提出的sysctl参数修改:<br />
<code class="highlighter-rouge">bash
#允许TCP/UDP打开的本地端口范围
echo "1024 65000" > /proc/sys/net/ipv4/ip_local_port_range
#内存脏数据回收参数,2.6内核中没有……
echo "100 1200 128 512 15 5000 500 1884 2">/proc/sys/vm/bdflush
#禁止ICMP ECHO响应
echo "1" > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
#记录伪装广播帧响应日志
echo "1" > /proc/sys/net/ipv4/icmp_ignore_bogus_error_responses
#允许转发
echo "1" > /proc/sys/net/ipv4/ip_forward
#可用的共享内存总量(bytes)
echo "268435456" >/proc/sys/kernel/shmall
#内核允许的最大共享内存段大小(bytes)
echo "536870912" >/proc/sys/kernel/shmmax
#链接跟踪库允许的最大值
echo "1048576" > /proc/sys/net/ipv4/netfilter/ip_conntrack_max
#TCP链接established状态的超时时间,同理,还有wait/close/last_ack等的超时时间设定
echo "600" > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
#决定检查一次相邻层记录的有效性的周期
echo "240" > /proc/sys/net/ipv4/neigh/default/gc_stale_time
#存在于ARP高速缓存中的最少层数,如果少于这个数,垃圾收集器将不会运行
echo "1024" > /proc/sys/net/ipv4/neigh/default/gc_thresh1
#保存在 ARP 高速缓存中的最多的记录软限制。垃圾收集器在开始收集前,允许记录数超过这个数字 5 秒
echo "2048" > /proc/sys/net/ipv4/neigh/default/gc_thresh2
#保存在 ARP 高速缓存中的最多记录的硬限制,一旦高速缓存中的数目高于此,垃圾收集器将马上运行
echo "4096" > /proc/sys/net/ipv4/neigh/default/gc_thresh3
#路由缓存最大值
echo "52428800" > /proc/sys/net/ipv4/route/max_size
#允许中继网络中的arp包
echo "1" > /proc/sys/net/ipv4/conf/all/proxy_arp
#TCP/IP会话的滑动窗口大小可变
echo "1" > /proc/sys/net/ipv4/tcp_window_scaling
</code><br />
最直接看到的两个调整,就是ip_conntrack_max和ip_conntrack_tcp_timeout_established了。<br />
而gc_*四个参数,是修改内核维护ARP表的参数,当arp -an|wc -l大于300的话,就需要修改这些参数,不然会出现“neighbour table overflow”或者“kernel: printk: 24 messages suppressed”这样的syslog,导致服务器ssh、ping无响应!</p>
squid加载tcmalloc性能优化测试(编译)
2010-02-04T00:00:00+08:00
squid
squid
tcmalloc
http://chenlinux.com/2010/02/04/performance-optimization-test-of-tcmalloc-loaded-by-squid-3
<p>话接上文,同一组LVS下,昨天采用重编译方式部署了另一台squid服务器。同样跑上一天,再次对比一番。今天流量比上回稍微少些,未加载的服务器cacti监控截图如下:<br />
<img src="/images/uploads/62d80b5eh73140310b076690.jpg" alt="" title="cpu" width="573" height="259" class="alignnone size-full wp-image-2565" /><br />
CPU占用率<br />
=====================<br />
<img src="/images/uploads/62d80b5eh73140316c9a2690.jpg" alt="" title="loadavg" width="569" height="254" class="alignnone size-full wp-image-2567" /><br />
负载</p>
<hr />
<p>同时,重编译过后的squid服务器监控截图如下:<br />
<img src="/images/uploads/62d80b5eh7314031cd5b3690.jpg" alt="" title="cpu-new" width="572" height="261" class="alignnone size-full wp-image-2568" /><br />
CPU占用率(那个尖峰是我运行了一下lsof确认是否加载,lsof这个命令真是大杀器,少用……)<br />
=====================<br />
<img src="/images/uploads/62d80b5eh731403230ce5690.jpg" alt="" title="loadavg-new" width="570" height="258" class="alignnone size-full wp-image-2569" /><br />
负载</p>
<p>这么一对比,效果马上就出来了。难怪官方建议要用编译加载的方式呢~~~</p>
squid加载tcmalloc性能优化测试(动态)
2010-02-01T00:00:00+08:00
squid
squid
tcmalloc
http://chenlinux.com/2010/02/01/performance-optimization-test-of-tcmalloc-loaded-by-squid-2
<p>昨天下午在一台squid上加载了tcmalloc。运行到现在,整整一天时间。现取LVS下与其完全相同配置的另一台未加载tcmalloc的squid服务器进行比较。<br />
环境说明:<br />
CPUinfo:Intel(R) Xeon(R) CPU E5405 @ 2.00GHz<br />
MEM:4G<br />
SQUID:Version 2.6.STABLE21<br />
当单台流量20M,TCP连接数6w时,未加载tcmalloc的服务器CPU占用率和负载情况如下图:<br />
<img src="/images/uploads/62d80b5eh730dca4ebd76690.jpg" alt="" title="cpu%" width="572" height="263" class="alignnone size-full wp-image-2559" /><br />
CPU占用率<br />
<img src="/images/uploads/62d80b5eh730dca57d6ee690.jpg" alt="" title="loadavg" width="575" height="257" class="alignnone size-full wp-image-2561" /><br />
负载</p>
<hr />
<p>而同时,加载了tcmalloc的服务器CPU占用率和负载情况如下图:<br />
<img src="/images/uploads/62d80b5eh730dca5f8985690.jpg" alt="" title="cpu%-new" width="577" height="263" class="alignnone size-full wp-image-2562" /><br />
CPU占用率<br />
<img src="/images/uploads/62d80b5eh730dca671bb7690.jpg" alt="" title="loadavg-new" width="579" height="259" class="alignnone size-full wp-image-2563" /><br />
负载(上一个尖峰就是我加载tcmalloc的时候)</p>
<p>从图中来看,加载tcmalloc,确实对squid处理高并发小图片请求有一定的性能优化帮助,但其本身对系统资源又有一定的耗用,导致负载反而略微提高(CPU占用率中sys也变高了)。而在并发量不大的时候,加载tcmalloc占用的CPU资源在图中也有显现。</p>
<p>或许这就是官方不建议采用动态加载方式的原因?下一步试验,采用重编译方式测试。</p>
squid几个第三方工具
2010-01-31T00:00:00+08:00
squid
http://chenlinux.com/2010/01/31/third-party-tools-about-squid
<p><a href="http://gadmintools.flippedweb.com/index.php?option=com_content&task=view&id=47&Itemid=37#Download">GADMIN-SQUID</a> 在linux环境下运行,经本人下载src后看,只能用于本机squid管理,不支持批量……</p>
<p><a href="http://www.zerozone.it/Software/Linux/SquidTL/">SQUIDTL</a> 针对IP、domain进行访问时长控制,C语言编写,需配合MySQL,好复杂的架构……</p>
<p><a href="http://sourceforge.net/projects/squidmodel/">SQUIDMODEL</a> java平台,俺还没搞定,未知其所以然</p>
<p><a href="http://cachevideos.com">CACHEVIDEO</a> 专门针对yotube等网站的视频文件缓存开发的rewrite_program,python编写。可惜是要花钱滴~~</p>
squid加载tcmalloc性能优化测试(原理)
2010-01-31T00:00:00+08:00
squid
squid
tcmalloc
http://chenlinux.com/2010/01/31/performance-optimization-test-of-tcmalloc-loaded-by-squid-1
<p>TCMalloc(Thread-Caching Malloc)是google开发的开源工具──“<a href="http://code.google.com/p/google-perftools/" title="google-perftools">google-perftools</a>”中的成员。其作者宣称tcmalloc相对于glibc2.3 malloc(aka-ptmalloc2)在内存的分配上效率和速度有6倍的性能提高,tcmalloc的常用场景是用于加速MySQL,不过据Wikipedia的hacker(Domas Mituzas)说,tcmalloc不仅仅对MySQL起作用,对squid也同样起作用(网上也有很多人在nginx上启用tcmalloc了),不过目前squid并没有official way来使用tcmalloc。<br />
TCMalloc的实现原理和测试报告请见一篇文章:《<a href="http://shiningray.cn/tcmalloc-thread-caching-malloc.html">TCMalloc:线程缓存的Malloc</a>》<br />
那么让我们赶紧给squid加载上tcmalloc,提高cache服务器在高并发情况下的性能,降低系统负载吧。<br />
因为服务器是64位OS,所以要先安装libunwind库。libunwind库为基于64位CPU和操作系统的程序提供了基本的堆栈辗转开解功能,其中包括用于输出堆栈跟踪的API、用于以编程方式辗转开解堆栈的API以及支持C++异常处理机制的API。(又cp一句话,^=^)<br />
<code class="highlighter-rouge">bashwget http://download.savannah.gnu.org/releases/libunwind/libunwind-0.99.tar.gz
tar zxvf libunwind-0.99.tar.gz
cd libunwind-0.99/
CFLAGS=-fPIC ./configure
make CFLAGS=-fPIC
make CFLAGS=-fPIC install</code><br />
普通的./configure&&make&&make install可不行哟~<br />
然后开始安装tcmalloc:<br />
<code class="highlighter-rouge">bashwget http://google-perftools.googlecode.com/files/google-perftools-1.8.1.tar.gz
tar zxvf google-perftools-1.8.1.tar.gz
cd google-perftools-1.8.1/
./configure --disable-cpu-profiler --disable-heap-profiler --disable-heap-checker --enable-minimal --disable-dependency-tracking
make && make install</code><br />
然后配置动态链接库,因为是之前是默认安装,这里自然就是/usr/local/lib了。<br />
<code class="highlighter-rouge">bashecho "/usr/local/lib" > /etc/ld.so.conf.d/usr_local_lib.conf
/sbin/ldconfig</code><br />
然后给squid加载tcmalloc。官方推荐是重新编译:<br />
先./configure,然后vi src/Makefile,修改如下:<br />
<code class="highlighter-rouge">c
....
squid_LDADD =
-L../lib \
-ltcmalloc_minimal \
\
repl/libheap.a repl/liblru.a \
....
....
data_DATA =
mib.txt
LDADD
= -L../lib -lmiscutil -lpthread -lm -lbsd -lnsl -ltcmalloc_minimal
EXTRA_DIST =
....
</code><br />
保存,make&&make install,OK!<br />
如果不愿意重新编译,那么动态加载吧。在/home/squid/sbin/squid -s之前执行export<br />
LD_PRELOAD=/usr/local/lib/libtcmalloc_minimal.so就可以了。<br />
最后运行lsof -n | grep tcmalloc看看,如果有<br />
squid 3811 root mem REG 202,1 1364560 446102 /usr/local/lib/libtcmalloc.so.0.0.0<br />
squid 3813 squid mem REG 202,1 1364560 446102 /usr/local/lib/libtcmalloc.so.0.0.0<br />
这样的输出,加载成功。效果如何,跑上几天再说咯~<br />
另:<br />
据网上说,如果squid的configure参数中,有–with-large-files的话,是没法加载tcmalloc的。</p>
header_replace试验
2010-01-31T00:00:00+08:00
squid
http://chenlinux.com/2010/01/31/intro-header_replace
<p>在linuxtone论坛上,偶见一贴,说采用如下配置,browser就可以正常遵守originserver的过期设置,而且充分利用browser本身的缓存设置,对server来说就可以达到减少304请求的效果,从而提升机器性能,节省带宽云云……</p>
<p>header_access Age deny all<br />
header_replace Age 1</p>
<p>这样reqly_header就显示成:Cache-Control: max-age=1。<br />
(让我很郁闷的一点是squid.conf.default里提供的header_access配置条目不全……)</p>
<p>从之前的cache驻留时间系列里知道,在HTTP的header里,max-age的优先级别高于Expires[Cache-Control>Expires>refresh_pattern>Etag>Last-Modified]。如果这里把max-age改了,那哪里还有地方去控制过期呢?</p>
<p>我想,帖子里估计是写错了,试验一下,果然,其实修改后的header应该是:Age: 1。</p>
<p>这种情况,应该是为了防止Cache-Control: max-age=0的定义导致304的出现(网上看到过一个试验结果,squid最多最多能接受的刷新极限是64秒,我汗,这都有人测试~~)</p>
<p>既然Age永远到不了max-age的限定,自然max-age定义失效了;</p>
<p>接下来试验,refresh_pattern里的max和LM-fator算法是否起作用。结果让我很无语——<br />
配置如右:refresh_pattern .gif$ 2 5% 5 ignore-reload</p>
<p>Date: Sun, 31 Jan 2010 10:32:45 GMT<br />
Last-Modified: Mon, 17 Nov 2008 06:53:22 GMT</p>
<p>LM-fator过期时间应该是(10:32-6:53)*5%=11分钟。max为5分钟。<br />
直到18:50:41还是HIT,Age=1,过期设定失效!</p>
<p>于是我取消http_replace和refresh配置生效后,18:53:15再wget,结果依然HIT,Age=155!</p>
<p>可见,之前LM-fator算法中的说法不完整,squid并不是单按照Date和Last-Modified时间就强制过期,它还得根据Age去判定——这点帖子倒是说对了,完全交给源站控制——问题在于,源站的运维如果真能策划好了这个,又何必让CDN在前端折腾这个age呢?鸡肋呀~~</p>
<hr />
<p>UPDATE:<br />
今天有几个客户切去快网,我在测试时发现,Age就是都被改成了1。除了源站的定义以外,或者快网自认为其提供的刷新API功能很好很强大??</p>
防盗链二进阶(squid外部ACL)
2010-01-31T00:00:00+08:00
CDN
squid
http://chenlinux.com/2010/01/31/anti-hotlinking-by-external_acl
<p>服务器防盗链设置,从最简单的referer判断,到进阶的key+time生成md5值,应该说是比较可靠了,而还有一种防盗链方式,基于IP/COOKIE的,这个我没找到太多有用信息,似乎IIS有个相关插件?<br />
只看到一篇squid的相关文章,简单的举了个防盗链的例子,未必有效(因为把cookie做明文处理,相比md5加密实在是防君子不防小人)。倒是从中学习一下external_acl_type用法,对squid进阶一番罢~~<br />
首先按惯例,上权威:《squid中文权威指南》6.1.3章节和12.5章节。<br />
用法如下:<br />
external_acl_type name [options] FORMAT.. /path/to/helper<br />
[helper arguments..]<br />
options包括:ttl、negtive_ttl、children、concurrency、cache和grace;<br />
FORMAT包括:%LOGIN,%EXT_USER,%IDENT,%SRC,%SRCPORT,%DST,%PROTO,%PORT,%METHOD,%MYADDR,%MYPORT,%PATH,%USER_CERT,%USER_CERTCHAIN,%USER_CERT_xx,%USER_CA_xx,%{Header},%{Hdr:member},%{Hdr:;member},%ACL,%DATA。<br />
外部程序输出结果必须是OK或者ERR,不过可以再带上一些keyword,比如user/passwd,ERR的messages,access.log里记录的%ea等等。<br />
cookie防盗链举例squid/libexec/check_cookie.pl如下:<br />
<code class="highlighter-rouge">perl
#!/usr/bin/perl -w
# 这个脚本仅仅是验证了Cache这个cookie的存在,没有严格的校验其值。
# disable output buffering
$|=1;
while () {
chop;
$cookie=$_;
if( $cookie =~ /$COOKIE/i) {
print "OK\n";
} else {
print "ERR\n";
}
}
</code><br />
squid.conf配置如下:<br />
<code class="highlighter-rouge">squid
external_acl_type download children=15 %{Cookie} squid/libexec/check_cookie.pl
acl dl external download
acl filetype url_regex -i .wmv .wma .asf .asx .avi .mp3 .smi .rm .ram .rmvb .swf .mpg .mpeg .mov .zip .mid
http_access deny filetype !dl
</code><br />
回过头来,想到之前的squid_session一文中,也是用的这个外部ACL~~</p>
ubuntu 9.10下linuxqq经常挂掉的解决方案
2010-01-30T00:00:00+08:00
linux
http://chenlinux.com/2010/01/30/solution-of-linuxqq-die-on-ubuntu-9-10
<p>ubuntu 9.10下linuxqq(官方的QQ,八百年不更新的那个了)</p>
<p>sudo vim /usr/bin/qq<br />
增加<br />
<code class="highlighter-rouge">bash
#!/bin/sh
export GDK_NATIVE_WINDOWS=true
cd /usr/share/tencent/qq/
./qq
</code><br />
经试验正确无误。。。linuxqq不再挂。。。五四陈科学院小道报道</p>
<p>原创文章如转载,请注明:转载自五四陈科学院[<a href="http://www.54chen.com]/">http://www.54chen.com]</a><br />
本文链接: <br />
<a href="http://www.54chen.com/uncategorized/ubuntu-910-hangs-under-linuxqq-regular-solution.html">http://www.54chen.com/uncategorized/ubuntu-910-hangs-under-linuxqq-regular-solution.html</a></p>
限速进阶
2010-01-30T00:00:00+08:00
CDN
nginx
php
http://chenlinux.com/2010/01/30/limit-speed-in-nginx-lighttpd
<p>继续横向比较,之前有squid的限速配置,现在说nginx限速,nginx关于限速的模块有三个:HTTPLimitZoneModule、HTTPCoreModule和HTTPLimitReqModule,详见官方wiki,就不再贴链接了,能不能上天知道,真是地上芙蓉姐,天上绿坝娘呀……</p>
<h1 id="limitzone">limit_zone配置</h1>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">http</span> <span class="p">{</span>
<span class="kn">limit_zone</span> <span class="s">one</span> <span class="nv">$binary_remote_addr</span> <span class="mi">10m</span><span class="p">;</span> <span class="c1">#每个clientIP定义一个10m大的session容器,根据32bytes/session,可以处理320000个session;
</span> <span class="kn">server</span> <span class="p">{</span>
<span class="kn">location</span> <span class="n">/download/</span> <span class="p">{</span>
<span class="kn">limit_conn</span> <span class="s">one</span> <span class="mi">1</span><span class="p">;</span><span class="c1">#限制一个IP并发连接数为1;
</span> <span class="kn">limit_rate</span> <span class="mi">300k</span><span class="p">;</span><span class="c1">#限制每个连接数速率为300k;
</span> <span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>目前网上的情况来看,zone方式的限速用的比较多,或许是较早的版本就支持这个模块吧。</p>
<h1 id="limitreq">limit_req配置</h1>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">http</span> <span class="p">{</span>
<span class="kn">limit_req_zone</span> <span class="nv">$binary_remote_addr</span> <span class="s">zone=one:10m</span> <span class="s">rate=1r/s</span><span class="p">;</span><span class="c1">#限制速率
</span> <span class="kn">server</span> <span class="p">{</span>
<span class="kn">location</span> <span class="n">/search/</span> <span class="p">{</span>
<span class="kn">limit_req</span> <span class="s">zone=one </span> <span class="s">burst=5</span><span class="p">;</span><span class="c1">#限制并发处理数
</span> <span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>这个模块是nginx0.7.22以后加上的,目前网上基本没什么实际说明,单就这个定义方式来看,感觉和之前的方式也没什么区别,a*b=c,一个定义a和b,一个定义b和c……<br />
或许只能去wiki死磕英文才能搞清楚一点点吧~~</p>
<p>然后是lighttpd的限速:</p>
<p>lighttpd限速也是有两个方式,server.kbytes-per-second(对域名路径限制)和connection.kbytes-per-second(对单一连接限制),问题在于没有对并发连接数的限制(也就是上面我写的算式里的b)。而且是kbytes不是kbits,根据网友的说法,当期望限制在60Mb的时候,配置成:server.kbytes-per-second = 1800就差不多了……</p>
<h1 id="section">限速配置</h1>
<pre><code class="language-lighttpd">connection.kbytes-per-second = 30 #单个连接不能超过30KB/s
$HTTP["host"] == "download.linuxfly.org" {
server.name = "download.linuxfly.org"
server.document-root = "/var/www/html/iso"
accesslog.filename = "/var/log/lighttpd/iso-access.log"
$HTTP["url"] =~ "^/download/" {
dir-listing.activate = "enable"
server.kbytes-per-second = 100 #/download路径下可用的最大是100KB/s
}
server.kbytes-per-second = 200 #除/download路径外,该域可用的最大带宽是200KB/s
}
</code></pre>
<p>最后,不要忘记lighttpd的单线程程序,还要受限于linux系统的单线程打开文件数限制。</p>
<p>lighttpd官方,还提供一个plugin方式,见http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs:TrafficShaping。<br />
这个需要在php代码中定制header来配合lighttpd。总的来说,lighttpd限速功能是不咋的……<br />
<code class="highlighter-rouge">bash
#打补丁
wget http://redmine.lighttpd.net/attachments/697/mod_speed.c
mv mod_speed.c lighttpd_1.5/src/
cat >>Makefile.am
<<EOF
lib_LTLIBRARIES += mod_speed.la
mod_speed_la_SOURCES = mod_speed.c
mod_speed_la_LDFLAGS = -module -export-dynamic -avoid-version -no-undefined
mod_speed_la_LIBADD = $(common_libadd)
EOF
</code><br />
# plugin配置参数<br />
<code class="highlighter-rouge">lighttpd
speed.just-copy-header = "enable"
speed.use-request = "enable"
</code><br />
//php定制header代码//<br />
<code class="highlighter-rouge">php
<?php
header("X-LIGHTTPD-KBytes-per-second: 50");
header("X-Sendfile: /path/to/file");
?>
</code></p>
防盗链进阶(nginx、lighttpd和squid类比)
2010-01-30T00:00:00+08:00
CDN
perl
php
nginx
squid
lighttpd
http://chenlinux.com/2010/01/30/anti-hotlinking-in-nginx-lighttpd-squid
<p>在折腾squid的rewrite.pl时,参考的是公司原有的一个防盗链脚本。如下:<br />
<code class="highlighter-rouge">perl
#! /usr/bin/env perl
use strict;
use Digest::MD5 qw(md5_hex);
use POSIX qw(difftime mktime);
$| = 1;
my $errUrl = "http://www.test.com/no_such_url.html";
my $secret = "123456";
my $expired = 7200;
while () {
my ($uri, $client, $ident, $method) = split;
print "$errUrln" and next unless ($uri =~ m#^(http://w*.?test.com)/(d{4})(d{2})(d{2})(d{2})(d{2})/(w{32})(/.+.mp3)$#i);
my ($domain, $year, $mon, $mday, $hour, $min, $md5, $path) = ($1, $2, $3, $4, $5, $6, $7, $8);
print "$errUrl\n" and next if $year < 1990 or $mon == 0 or $mon > 12 or $mday == 0 or $mday > 31 or $hour > 23 or $min > 59;
print "$errUrl\n" and next if abs(difftime((int(time() / 100) * 100), mktime(00, $min, $hour, $mday, $mon - 1, $year - 1900))) > $expired;
$path =~ s#%(..)#pack("c", hex($1))#eg;
print "$errUrl\n" and next if $md5 ne md5_hex($secret . $year . $mon . $mday . $hour . $min . $path);
print $domain . $path, "\n";
}
</code><br />
今天在网上看到lighttpd相似的配置。lighttpd自带mod_secdownload模块实现这种防盗链方法。具体配置及php代码如下例(详见http://trac.lighttpd.net/trac/wiki/Docs%3AModSecDownload):<br />
<code class="highlighter-rouge">php
<?
$secret = "verysecret"; //加密字符串,必须跟lighttpd.conf里边保持一致
$uri_prefix = "/dl/"; //虚拟的路径,必须跟lighttpd.conf里边保持一致
# filename
$f = "/secret-file.txt"; //实际文件名,必须要加"/"斜杠
# current timestamp
$t = time();
$t_hex = sprintf("%08x", $t);
$m = md5($secret.$f.$t_hex);
# generate link
printf('%s', $uri_prefix, $m, $t_hex, $f, $f);
?>
</code><br />
lighttpd配置文件: <br />
server.modules = ( …, “mod_secdownload”, … )<br />
secdownload.secret = “verysecret”<br />
secdownload.document-root = “/home/www/servers/download-area/”<br />
secdownload.uri-prefix = “/dl/”<br />
secdownload.timeout = 10</p>
<p>nginx也有类似功能,不过不是自带,secure_link_module模块,打补丁需要重编译:</p>
<p>wget http://www.sbear.cn/wp-content/uploads/2009/09/nginx-secure-link-ttl.patch<br />
cd nginx-0.7.62<br />
patch -p1 < ../nginx-secure-link-ttl.patch<br />
./configure –with-http_secure_link_module<br />
……<br />
具体配置及php例子如下(详见http://wiki.nginx.org/NginxHttpSecureLinkModule):<br />
<code class="highlighter-rouge">nginx
location /down/ {
secure_link_secret "sbear.cn"; //密钥
secure_link_ttl on;
root /data/test/down;
if ($secure_link = "") {
return 403;
}
rewrite ^ /$secure_link break;
}
</code><br />
<code class="highlighter-rouge">php
<?php
define(URL_TIMEOUT, 3600); //这里设置过期时间单位是秒
$prefix = "<a href="http://www.sbear.cn/down&quot;;">http://www.sbear.cn/down";</a>
$protected_resource = "test.exe";
$secret = "sbear.cn"; //密钥
$time = pack('N', time() + URL_TIMEOUT);
$timeout = bin2hex($time);
$hashmac = md5( $protected_resource . $time . $secret );
$url = $prefix . "/" . $hashmac . $timeout . "/" . $protected_resource;
echo "down";
echo time();
?>
</code></p>
<p>那不打补丁,有什么防盗链的办法么?当然有。nginx和lighttpd都支持最简单的referer判断。</p>
<p>nginx有ngx_http_referer_module模块,和apache、squid一样可以rewrite,配置如下:<br />
<code class="highlighter-rouge">nginx
location ~* .(gif|jpg|png)$ {
valid_referers none blocked www.test.com baidu.com;
if ($invalid_referer) {
rewrite ^/ http://www.test.com/error.html;
}
}
</code></p>
<p>lighttpd配置如下:</p>
<p>$HTTP[“referer”] !~ “^(http://.<em>.(test.com|baidu.cn))”<br />
{<br />
$HTTP[“url”] =~ “.(jpg|jpeg|png|gif|rar|zip|mp3|swf|flv|wmv|rm|avi)$” {<br />
url.redirect = (“.</em>“ => http://www.test.com/”)<br />
}<br />
}<br />
不过还是那句话,这个功能破解起来确实太容易,呵呵~</p>
<p>除了上面说的NginxHttpSecureLinkModule,还有另一个模块ngx_http_accesskey_module,其工作原理是根据client的IP,加上配置定义的key,生成32位MD5值,然后进行匹配。详见http://wiki.codemongers.com/NginxHttpAccessKeyModule,不过我这居然打不开……只好详见网友博客了:<br />
wget <a href="http://wiki.nginx.org/File:Nginx-accesskey-2.0.3.tar.gz">http://wiki.nginx.org/File:Nginx-accesskey-2.0.3.tar.gz</a><br />
tar zxvf Nginx-accesskey-2.0.3.tar.gz<br />
sed -i ‘s/$HTTP_ACCESSKEY_MODULE/ngx_http_accesskey_module/g nginx-accesskey/config<br />
./configure –add-module=/path/to/nginx-accesskey<br />
#配置文件<br />
location /download {<br />
accesskey on;<br />
accesskey_hashmethod md5;<br />
accesskey_arg “key”;<br />
accesskey_signature “mypass$remote_addr”;<br />
}<br />
//php测试页面,$output_add_key正常,$output_org_url返回403//<br />
<?
$ipkey= md5("mypass".$_SERVER['REMOTE_ADDR']);
$output_add_key="<a href="http://www.example.cn/download/G3200507120520LM.rar?key=">http://www.example.cn/download/G3200507120520LM.rar?key="</a>.$ipkey.">download_add_key<br
/>";
$output_org_url="<a href=<a href="http://www.example.cn/download/G3200507120520LM.rar">http://www.example.cn/download/G3200507120520LM.rar</a>>download_org_path<br
/>";
echo $output_add_key;
echo $output_org_url;
?><br />
而另一个博客这么说:<br />
wget http://www.ieesee.net:8080/~uingei/nginx-accesskey-2.0.3.diff.bz2<br />
cd nginx-0.7.14<br />
bzcat ../nginx-accesskey-2.0.3.diff.bz2 | patch -p1<br />
./configure –with-http_accesskey_module …<br />
根据我的观察,这个应该是最初的办法。另,该博客说sec_link是nginx0.7.18后加的官方模块。</p>
日志管理
2010-01-29T00:00:00+08:00
linux
apache
nginx
logrotate
http://chenlinux.com/2010/01/29/intro-cronolog-logrotate
<p>第一个,最常见的cronolog,因为他配对apache,apache太常见,所以cronolog也就常见了~~稳定版本1.6.2,似乎不支持2G以上的日志,因为那时候linux内核也不支持。。。现在有beta版可以,另据网友分析,其实只要修改1.6.2源代码中的openfile函数成openfile64就行了……<br />
使用方法:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>CustomLog "|/usr/local/sbin/cronolog /var/log/httpd/www/access%Y%m%d.log" combined
</code></pre>
</div>
<p>其实就是利用管道,传递给cronolog记录日志。<br />
对于不支持在配置文件里利用管道的,也有别的变通办法,比如命名管道。像nginx日志就能这么做:</p>
<p>mkfifo /path/to/nginx/logs/access_log_pipe<br />
/path/to/cronolog /path/to/log/access_%Y%m%d.log /path/to/nginx/logs/access_log_pipe &</p>
<p>然后编辑nginx.conf,修改如下:</p>
<p>access_log /path/to/nginx/logs/access_log_pipe combined</p>
<p>最后要注意的是,必须在启动nginx之前,先启动cronolog。</p>
<p>第二个,系统自带的logrotate,一般系统本身的syslog、crond、yum等,都是用它。对于不必要保存所有的日志,采用这个回滚程式正当其时。/etc/logrotate.conf默认配置不算太复杂,还能使用include,不过其全部参数真是不少,如下表:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>参数 功能
compress 通过gzip 压缩转储以后的日志
nocompress 不需要压缩时,用这个参数
copytruncate 用于还在打开中的日志文件,把当前日志备份并截断
nocopytruncate 备份日志文件但是不截断
create mode owner group 转储文件,使用指定的文件模式创建新的日志文件
nocreate 不建立新的日志文件
delaycompress 和 compress 一起使用时,转储的日志文件到下一次转储时才压缩
nodelaycompress 覆盖 delaycompress 选项,转储同时压缩。
errors address 专储时的错误信息发送到指定的Email 地址
ifempty 即使是空文件也转储,这个是 logrotate 的缺省选项。
notifempty 如果是空文件的话,不转储
mail address 把转储的日志文件发送到指定的E-mail 地址
nomail 转储时不发送日志文件
olddir directory 转储后的日志文件放入指定的目录,必须和当前日志文件在同一个文件系统
noolddir 转储后的日志文件和当前日志文件放在同一个目录下
prerotate/endscript 在转储以前需要执行的命令可以放入这个对,这两个关键字必须单独成行
postrotate/endscript 在转储以后需要执行的命令可以放入这个对,这两个关键字必须单独成行
daily 指定转储周期为每天
weekly 指定转储周期为每周
monthly 指定转储周期为每月
rotate count 指定日志文件删除之前转储的次数,0 指没有备份,5 指保留5 个备份
tabootext [+] list 让logrotate 不转储指定扩展名的文件,缺省的扩展名是:.rpm-orig,
.rpmsave, v, 和 ~
size size 当日志文件到达指定的大小时才转储,Size可以指定bytes(缺省)以及KB(sizek)或者MB (sizem).
</code></pre>
</div>
<p>第三个,newsyslog,这个东西感觉是logrotate的兄弟版,因为其配置文件写法都是采用{}的方式,而且命名也这么针锋相对的。习惯了用logrotate的,对于不能回滚而要求分割保留的,可以试试这个。</p>
<p>logrotate的写法举例:</p>
<p>/path/to/wtmp{<br />
daily<br />
minsize 5M<br />
create 0644 root utmp<br />
rotate 1<br />
}<br />
newsyslog的写法举例:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>set squid_logpath = /usr/local/squid/var/logs
set squid_log = /usr/local/squid/var/logs/access.log
set date_squid_log = /usr/local/squid/var/logs/access%Y%M%D.log
SQUID{
restart: run
/usr/local/squid/sbin/squid -k rotate
log: SQUID
squid_log squid squid 644
archive:
SQUID date_squid_log 0
}
</code></pre>
</div>
<p>第四个,也是万能的一个,嗯,就是自己写脚本,定时gzip……</p>
端口转发
2010-01-28T00:00:00+08:00
linux
iptables
perl
http://chenlinux.com/2010/01/28/transparent-by-ports
<p>常见的linux端口转发,是iptables方式,方法如下:<br />
<code class="highlighter-rouge">bash
#modprobe iptable_nat
#modprobe ip_conntrack
#service iptables stop
#echo 1 > /proc/sys/net/ipv4/ip_forward
#iptables -t nat -I PREROUTING -p tcp --dport 443 -j DNAT --to 1.2.3.4
#iptables -t nat -I POSTROUTING -p tcp --dport 8081 -j MASQUERADE
#service iptables save
#service iptables start
</code><br />
其次网上有不少推荐rinetd,方法如下:<br />
<code class="highlighter-rouge">bash
#wget http://www.boutell.com/rinetd/http/rinetd.tar.gz
#tar zxvf rinetd.tar.gz
#cd rinted
#./configure && make && make install
# cat /etc/rinetd.conf
0.0.0.0 443 www.test.com 443 allow *.*.*.*
logfile /var/log/rinetd.log
</code><br />
看起来比iptables有个优势,可以采用域名解析,可惜做不到根据域名的不同,把相同端口的请求转向不同IP,其实也就跟直接写IP没多少区别了。<br />
第三还有ssh的端口转发,其实是把原TCP端口的数据转由ssh通道传输。方法如下:<br />
本地转发:ssh -g -L <local port="">:<remote host="">:
远程转发:ssh -g -R <local port="">:<remote host="">:
不过客户未必(或许说基本不可能)给外人开放ssh吧~~
第四,socket网络编程实现。我在网上发现了相关的perl脚本。
例一代码如下:
```perl
#!C:Perlbinperl.exe
#端口重定向(fork、IO::Select)
#By shanleiguang@gmail.com, 2005/10
use strict;
use warnings;
use IO::Socket;
use IO::Select;
use Getopt::Std;
use POSIX qw(:sys_wait_h strftime);
use constant FOREVER => 1;
use constant BUFSIZE => 4096;
#父进程下处理僵死子进程
sub zombie_reaper {
while(waitpid(-1, WNOHANG) > 0) {}
$SIG{CHLD} = &zombie_reaper;
}
$SIG{CHLD} = &zombie_reaper;
#处理参数
my %opts;
getopts('h:l:t:p:', %opts);
print_help() and exit if(defined($opts{'h'}));
print_help() and exit if(not defined($opts{'t'}) or not defined($opts{'p'}));
print_help() and exit if($opts{'t'} !~ m/^d+.d+.d+.d+$/);
print_help() and exit if($opts{'l'} !~ m/^d+$/);
print_help() and exit if($opts{'p'} !~ m/^d+$/);
my $listen_port = (defined($opts{'l'})) ? $opts{'l'} : 8080;
my $target_ip = $opts{'t'};
my $target_port = $opts{'p'};
#在本地创建监听Socket
my $socket_listen = IO::Socket::INET->new(
LocalPort => $listen_port,
Proto => 'tcp',
Listen => 5,
Reuse => 1,
);
print timestamp(), ", listening on port $listen_port ...n";
#新建两个用于Socket IO监视的IO::Select对象
my $readers = IO::Select->new();
my $writers = IO::Select->new();
$readers->add($socket_listen);
#父进程仅监视Listening Socket的accept事件
while(FOREVER) {
my @readers = $readers->can_read;
foreach my $reader (@readers) {
if($reader eq $socket_listen) {
#创建子进程处理后续的转发,父进程继续监视Listening Socket
fork and next;
my $socket_client = $socket_listen->accept();
#在子进程中,不再需要对Listening Socket进行操作
$readers->remove($socket_listen);
$socket_listen->close();
#子进程
as_server($socket_client);
exit;
}
}
}
#子进程
sub as_server {
my $socket_client = shift;
my $client_port = $socket_client->peerport();
my $client_ip = $socket_client->peerhost();
#创建到目标地址:端口的Socket连接
my $socket_forward = IO::Socket::INET->new(
PeerAddr => $target_ip,
PeerPort => $target_port
);
print timestamp(), ", $client_ip:$client_port$target_ip:$target_port.n";
#监视socket_client、socket_forward的IO情况
$readers->add($socket_client);
$readers->add($socket_forward);
$writers->add($socket_client);
$writers->add($socket_forward);
my ($rbuffer_forward, $rbuffer_client) = ('', '');
while(FOREVER) {
my @readers = $readers->can_read;
foreach my $reader (@readers) {
my $rbuffer;
#当socket_client可读时,将读取的内容追加到rbuffer_client后
#假如读取失败,则退出子进程
if($reader eq $socket_client) {
exit if(not recv($reader, $rbuffer, BUFSIZE, 0));
$rbuffer_client.= $rbuffer;
}
#当socket_forward可读时,将读取的内容追加到rbuffer_forward后
#假如读取失败,则退出子进程
if($reader eq $socket_forward) {
exit if(not recv($reader, $rbuffer, BUFSIZE, 0));
$rbuffer_forward .= $rbuffer;
}
}
my @writers = $writers->can_write;
foreach my $writer (@writers) {
#当socket_client可写,且rbuffer_forward不为空时,将rbuffer_forward
#内容写入socket_client,假如写失败,则退出子进程
if($writer eq $socket_client) {
next if(not $rbuffer_forward);
exit if(not send($writer, $rbuffer_forward, 0));
$rbuffer_forward = '';
}
#当socket_forward可写,且rbuffer_client不为空时,将rbuffer_client
#内容写入socket_forward,假如写失败,则退出子进程
if($writer eq $socket_forward) {
next if(not $rbuffer_client);
exit if(not send($writer, $rbuffer_client, 0));
$rbuffer_client = '';
}
}
}
}
sub timestamp {
return strftime "[%y/%m/%d,%H:%M:%S]", localtime;
}
sub print_help {
my $filename = (split /\/, $0)[-1];
print <<help>>>
$filename [-h,-l:]
-h print help
-l listening local port, default 8080
-t target ipaddr
-p target port
By shanleiguang@gmail.com, 2005/10
HELP
}
```
例二代码如下:
```perl
#!C:Perlbinperl.exe
#端口重定向(POE)
#By shanleiguang@gmail.com, 2005/10
#POE结构:
#
#Driver->Filter->Wheel->Components
#
|______|______|________|
#
|
# Session
#
|
# Kernel
#
#Driver: 底层文件操作的抽象,在编程时不会直接用到
#Filter: 底层、中层协议操作的抽象,通常不会直接用到
#Wheel: 高层协议操作的抽象,经常要用到
#Components:POE提供的一些拿来就能用的组件
#Session: 会话抽象,会话中需要创建高层协议抽象
#Kernel: POE管理会话的内核
#
#POE对象的数据结构:
#
#$_[HEAP]:是会话唯一的数据存储区;
#$_[SESSION]:是指向会话自身的引用;
#$_[KERNEL]:是指向会话管理内核的引用;
#@_[ARG0..ARG9]:用于传递给各事件处理函数的参数;
#
#还是实例最直观:
#在父会话中创建一个监听用的Socket,当有客户端连接,即有accept_sucess事件发生时,
#则创建一个子会话处理后续事件,并将accept获得的客户端Socket传递给子会话;子会话
#创建到目标的Socket,连接过程中,如果客户端Socket中有input事件,则将客户端的input
#内容缓存在一个队列中,当连接成功后,发送给到目标的那个Socket中,见下图:
#
# +-------------------------------+
# /| Socket_listen |
# / +-------------------------------+
#Client|Socket_client Socket_server|Target
# +-------------------------------+
# Forwarder
use strict;
use warnings;
use Socket;
use Getopt::Std;
use POSIX qw(strftime);
use POE qw(
Wheel::SocketFactory
Wheel::ReadWrite
Filter::Stream
);
#Get Options
my %opts;
getopts('hl:t:p:', %opts);
print_help() and exit if(defined($opts{'h'}));
print_help() and exit if(not defined($opts{'t'}) or not defined($opts{'p'}));
print_help() and exit if($opts{'t'} !~ m/^d+.d+.d+.d+$/);
print_hekp() and exit if($opts{'p'} !~ m/^d+$/);
my $listen_port = (defined($opts{'l'})) ? $opts{'l'} : 8080;
my $target_addr = $opts{'t'};
my $target_port = $opts{'p'};
###Create Parent - 'Listen Session'###
###创建父会话用于监听客户端的连接
###会话创建的最后将进入_start状态,执行_start的handler
###accept_success即在_start的handler中创建监听Socket的Wheel中
###的SuccessEvent事件,它的handler是forwarder_create函数
###$_[ARG0]是wheel::SocketFatory的SuccessEvent传递的参数
POE::Session->create(
inline_states => {
_start => &forwarder_server_start,
_stop => sub { print timestamp(), ", forwarder server stopped."; },
accept_success => sub { &forwarder_create($_[ARG0]); },
accept_failure => sub { delete $_[HEAP]->{server_wheel} },
},
);
$poe_kernel->run();
exit;
###Event handlers for Parent Session###
###父会话中的事件处理函数
sub forwarder_server_start {
print timestamp(), ", listening on port $listen_port and ";
print "forward to $target_addr:$target_port\n";
#在父会话的存储区创建一个监听Socket类型的Wheel
$_[HEAP]->{server_wheel} = POE::Wheel::SocketFactory->new(
BindPort => $listen_port,
SocketProtocol => 'tcp',
ListenQueue => SOMAXCONN,
Reuse => 'on',
#ARG0 of SuccessEvent
SuccessEvent => 'accept_success',
FailureEvent => 'accept_failure',
);
}
###Create Child - 'Forward Session'###
###创建子会话
sub forwarder_create {
my $socket = shift;
POE::Session->create(
inline_states => {
_start => &forwarder_start,
_stop => sub {
print ' ' x 4, timestamp(), ', sessionId:';
print $_[SESSION]->ID, ", forwarder stop\n";
},
client_input => &client_input,
client_error => sub {
delete $_[HEAP]->{wheel_client};
delete $_[HEAP]->{wheel_server};
},
server_connect => &server_connect,
server_input => sub {
$_[HEAP]->{wheel_client}->put($_[ARG0]) if(exists $_[HEAP]->{wheel_client});
},
server_error => sub {
delete $_[HEAP]->{wheel_client};
delete $_[HEAP]->{wheel_server};
},
},
#Parameters to '_start' Event
args => [$socket],
);
}
##Event Handlers of Child Session##
sub forwarder_start {
my ($heap, $socket) = @_[HEAP, ARG0];
print ' ' x 4, timestamp(), ', sessionId:';
print $_[SESSION]->ID, ", forwarder startn";
#Buffer client's input while connecting to the target
$heap->{state} = 'connecting';
$heap->{queue} = [];
#ClientForwarder server
$heap->{wheel_client} = POE::Wheel::ReadWrite->new(
Handle => $socket,
Driver => POE::Driver::SysRW->new(),
Filter => POE::Filter::Stream->new(),
InputEvent => 'client_input',
ErrorEvent => 'client_error',
);
#Forwarder servertarget
$heap->{wheel_server} = POE::Wheel::SocketFactory->new(
RemoteAddress => $target_addr,
RemotePort => $target_port,
SuccessEvent => 'server_connect',
FailureEvent => 'server_error',
);
}
sub server_connect {
my ($kernel, $session, $heap, $socket) = @_[KERNEL, SESSION, HEAP, ARG0];
#Replace
$heap->{wheel_server}
$heap->{wheel_server}
= POE::Wheel::ReadWrite->new(
Handle => $socket,
Driver => POE::Driver::SysRW->new,
Filter => POE::Filter::Stream->new,
InputEvent => 'server_input',
ErrorEvent => 'server_error',
);
$heap->{state} = 'connected';
$kernel->call($session, 'client_input', $_) foreach(@{$heap->{queue}});
$heap->{queue} = [];
}
sub client_input {
my ($heap, $input) = @_[HEAP, ARG0];
push @{$heap->{queue}}, $input and return if($heap->{state} eq 'connecting');
$heap->{wheel_server}->put($input) if(exists $heap->{wheel_server});
}
#Common subroutines
sub timestamp {
return strftime "[%H:%M:%S]", localtime;
}
sub print_help {
my $filename = (split /\/, $0)[-1];
print <<help>>>
$filename [-h,-l:]
-h print help
-l listen port
-t target ipaddress
-p target port
A simple TCP forwarder server, 2005/10
By shanleiguang@gmail.com
HELP
}
```
使用方法:
# perl tcpForwarder.pl -l 8080 -t
xxx.xxx.xxx.xxx -p 80
[xx:xx:xx:], listening on local port 8080 and forward to
xxx.xxx.xxx.xxx:80...
...</help></help></remote></local></remote></local></p>
squid页面跳转试验
2010-01-27T00:00:00+08:00
squid
http://chenlinux.com/2010/01/27/test-location-in-squid
<p>某客户页面跳转试验记录</p>
<h2 id="section">一、客户需求</h2>
<p>某客户加速域名下某路径http://www.test.com/zhlc/不想让网民直接访问,希望当有直接访问该路径的请求时,都跳转到search.test.com域名下相应的路径去。</p>
<h2 id="section-1">二、解决思路</h2>
<p>a) 客户源站增添rewrite配置进行处理,将该类请求url改写成所期望的url;<br />
b) Squid端对请求进行处理,将该类请求url改写成所期望的url。<br />
根据售前沟通结果,这个处理步骤交由squid完成。</p>
<h2 id="squidrequest">三、Squid对request的处理流程(略,见前博文)</h2>
<h2 id="section-2">四、两种跳转方法的介绍与比较</h2>
<p>针对跳转需求,提供两个两种方法:<br />
a) Squid支持调用外部程序脚本改写url,即redirect_program(2.6以上版本为url_rewrite_program)。所有的url请求,在squid处理流程的第三步,都会检查rewrite_program,然后将改写后的结果推入流程继续进行(为了提高运行速度,降低服务器负载,可以采用url_rewrite_access控制请求和redirector_bypass在繁忙时进行透传)。<br />
根据《squid中文权威指南》11章的介绍,squid传递给redirect_proram的流格式为:<br />
URL IP/FQDN IDENT METHOD<br />
Rewrite处理时,也就是对这四个字段进行处理,一般地说,也就处理第一个字段——$url。<br />
《squid中文权威指南》中提供了标准的perl脚本,演示了处理办法。我们测试脚本时,只需要创建一个文本文件写出url,对文件执行脚本即可。某客户测试过程如下示,为说明方便,也使用了其他shell命令做比较:<br />
<code class="highlighter-rouge">bash
[root@tinysquid1 etc]# cat testurl.lst
http://www.test.com/zhlc/
http://www.test.com/zhlc/images/1.jpg
http://www.test.com/zhlc/index.html
http://www.test.com/zhlc/index.aspx?oid=1
http://www.test.com/zhlc/index.aspx?oid=1&pid=2
[root@tinysquid1 etc]# cat testurl.lst |sed s/www/search/
http://search.test.com/zhlc/
http://search.test.com/zhlc/images/1.jpg
http://search.test.com/zhlc/index.html
http://search.test.com/zhlc/index.aspx?oid=1
http://search.test.com/zhlc/index.aspx?oid=1&pid=2
[root@tinysquid1 etc]# cat redirector.awk
#!/bin/awk -f
{
split($1,myarray,".test.com/")
}
{
myarray[1]=http://search";
print myarray[1]".test.com/"myarray[2]"\n"
}
[root@tinysquid1 etc]# ./redirector.awk testurl.lst
http://search.test.com/zhlc/
http://search.test.com/zhlc/images/1.jpg
http://search.test.com/zhlc/index.html
http://search.test.com/zhlc/index.aspx?oid=1
http://search.test.com/zhlc/index.aspx?oid=1&pid=2
[root@tinysquid1 etc]# cat redirector.pl
#!/usr/bin/perl -wl
use strict;
$|=1;
while () {
my ($url,$client,$ident,$method) = ();
($url, $client, $ident, $method) = split;
if ($url =~ m#^http://www.test.com/zhlc/#) {
$url=~ s/www/search/;
print "$url\n";
} else {
print "$urln";
}
}
[root@tinysquid1 etc]# ./redirector.pl testurl.lst
http://search.test.com/zhlc/
http://search.test.com/zhlc/images/1.jpg
http://search.test.com/zhlc/index.html
http://search.test.com/zhlc/index.aspx?oid=1
http://search.test.com/zhlc/index.aspx?oid=1&pid=2
</code></p>
<p>不过虽然awk也是流处理,但作为squid的外挂program测试却没法真起作用。所以只能用perl(网上看到也有用php和python的)。</p>
<p>采用rewrite方法后,squid日志记录如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1264603532.881 1147 59.151.*.* TCP_MISS/200 86230 GET http://www.test.com/zhlc/images/header.jpg - DIRECT/1.2.3.173 image/jpeg "http://www.test.com/zhlc/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQPinyinSetup 620; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)"
</code></pre>
</div>
<p>虽然http_status显示的200,不过www.test.com的源站IP为1.2.3.172,search.test.com的源站IP为1.2.3.173,可见squid已经将请求转发给search.test.com了。</p>
<p>由上面的流程可知,squid是先检查acl,然后rewrite,再检查HIT/MISS,所以当设定了search.test.com/.<em>的 cache_deny 后,所有www.test.com/zhlc/.</em>的重复访问也都是MISS。</p>
<p>b) 第二种方法,不属于专门的跳转重定向,算是个妙用吧:</p>
<p>Squid为了美观方便,提供了一些错误信息的定制功能。之前的源站错误跳转,就是修改了ERROR_DIRECTORY里html的meta标签做的。除此以外,针对error_diretory里的ACCESS_DENIED页面,还有专门的另一个configure参数进行定制——deny_info。其用法如下:</p>
<pre><code class="language-squid">acl test url_regex -i ^http://www.test.com/zhlc/.*
http_access deny test
deny_info http://search.test.com/zhlc/ test
</code></pre>
<p>如果需要显示的信息已经编辑在error_diretory里了,那就可以直接写文件名而不用写url。Squid.conf.default中举例是 <code class="highlighter-rouge">deny_info ERR_CUSTOM_ACCESS_DENIED bad_guys</code>。</p>
<p>这个deny_info,常用的地方是防盗链。在deny盗链的同时,加上一个源站的logo图片url,正好让盗链网站替自己做宣传~~</p>
<p>采用deny_info方法后,squid日志记录如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1264661272.822 2 59.151.*.* TCP_DENIED/302 320 GET http://www.test.com/zhlc/ - NONE/- text/html "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQPinyinSetup 620; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)"
1264661273.753 854 59.151.*.* TCP_MISS/200 9166 GET http://search.test.com/zhlc/ - DIRECT/1.2.3.173 text/html "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQPinyinSetup 620; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)"
</code></pre>
</div>
<p>显示的是TCP_DENIED/302。</p>
<p>c) 两个方法比较</p>
<p>最表面可见的不同,就是当IE访问http://www.test.com/zhlc/时,第一种方法地址栏依然显示(httpwatch也一样)http://www.test.com/zhlc/,而第二种方法也显示为http://search.test.com/zhlc/了。</p>
<p>根据之前论述可知,第一种访问时,clinet提出第一个http://www.test.com/zhlc/请求,squid在流程第三步改写url,从search源站取回html代码,查询到相关资源(即http://www.test.com/zhlc/.*),然后重复请求过程,逐一改写,从search源站取回所有文件;</p>
<p>第二种访问时,client提出第一个http://www.test.com/zhlc/请求,squid在流程第二步检查出相匹配的acl规则,返回deny信息给client,即请IE浏览器显示http://search.test.com/zhlc/,然后client重新开始一次链接建立过程,经dns解析等步骤,连上search.test.com的,取回http://search.test.com/zhlc/.*。</p>
<h2 id="section-3">五、客户页面分析</h2>
<p>在日志分析过程中,还发现一个问题,当squid首先从search.test.com源站取回html代码后,为什么调用的相关页面资源url请求也都是www.test.com的呢?看http://www.test.com/zhlc/页面代码,发现如下:<br />
```html</p>
<link href="/zhlc/style/reset.css" rel="stylesheet" type="text/css" />
<link href="/zhlc/style/main.css" rel="stylesheet" type="text/css" />
<script src="/zhlc/Scripts/AC_RunActiveContent.js" type="text/javascript">
<a href="#"><img src="/zhlc/images/logo.jpg" width="154" height="80" border="0"/></a>
……
```
其页面代码都使用了相对路径,所以才导致了从search端取回的代码依然调用www的url的现象。
## 六、总结
某客户页面跳转试验至此,因为squid处理流程和客户源站代码等多方面原因,只能采用统一跳转至http://search.test.com/zhlc/单一页面的方法,确保客户在IE地址栏里看到有跳转的效果……
最后,总结试验中的两种办法,其改写过程,可以归纳成rewrite是squid-origin过程的,deny_info是squid-client过程的。
</script>
CUshell版惊见中国特色脚本
2010-01-24T00:00:00+08:00
bash
http://chenlinux.com/2010/01/24/shell-script-for-train-setting
<p>58同城:<br />
<code class="highlighter-rouge">bash
#!/bin/bash
#该脚本仅用于58同城网站刷票
#到达目的地
DES="(天水|兰州|西安)"
#刷票url
URL=http://bj.58.com/huochepiao/a1/
OLDMD5=`curl $URL | egrep "$DES" | sed '$d' | md5sum | awk '{print $1}'`
while true; do
if [ "$OLDMD5" == `curl $URL | egrep "$DES" | sed '$d' | md5sum | awk '{print $1}'` ];then
sleep 300
else
MESSAGE=`curl $URL | egrep "$DES" | sed '$d' | head -n1`
SEND=`echo $MESSAGE | awk -F">" '{print $3}' | sed -e 's/<.*//'`
DATE=`echo $SEND | awk -F":" '{print $2}'`
#判断DATE变量,使其匹配你需要出发的日期
if [ "$DATE" == "2010-02-08" -o "$DATE" == "2010-02-09" -o "$DATE" == "2010-02-10" -o "$DATE" == "2010-02-11" -o "$DATE" == "2010-02-12" -o "$DATE" == "2010-01-28" ];then
#输入需要发送信息的命令,飞信或mail等.....,发送的内容为$SEND变量
/usr/bin/curl -d cdkey=xxxx-xxx-xxxx-xxxxx -d password=xxxxxx -d addserial=xxx -d phone=13000000000 -d message="$SEND:58.com" http://sdkhttp.eucp.b2m.cn/sdkproxy/asynsendsms.action
fi
OLDMD5=`curl $URL | egrep "$DES" | sed '$d' | md5sum | awk '{print $1}'`
sleep 300
fi
done
</code><br />
赶集网:<br />
<code class="highlighter-rouge">bash
#!/bin/bash
#该脚本仅用于赶集网刷票
#到达目的地
DES="(天水|兰州|西安)"
#刷票url
URL=http://bj.ganji.com/piao/sell/
OLDMD5=`curl $URL | egrep "$DES" | md5sum | awk '{print $1}'`
while true; do
if [ "$OLDMD5" == `curl $URL | egrep "$DES" | md5sum | awk '{print $1}'` ];then
sleep 300
else
MESSAGE=`curl $URL | egrep "$DES" | head -n1`
SEND=`echo $MESSAGE | awk -F">" '{print $3}' | sed -e 's/<.*//'`
DATE=`echo $SEND | awk -F":" '{print $2}'`
#判断DATE变量,使其匹配你需要出发的日期
if [ "$DATE" == "02-08" -o "$DATE" == "02-09" -o "$DATE" == "02-10" -o "$DATE" == "02-11" -o "$DATE" == "02-12" -o "$DATE" == "01-28" ];then
#输入需要发送信息的命令,飞信或mail等.....,发送的内容为$SEND变量
/usr/bin/curl -d cdkey=xxxx-xxx-xxxx-xxxxx -d password=xxxxxx -d addserial=xxx -d phone=13000000000 -d message="$SEND:ganji.com" <a href="http://sdkhttp.eucp.b2m.cn/sdkproxy/asynsendsms.action">http://sdkhttp.eucp.b2m.cn/sdkproxy/asynsendsms.action</a>
fi
OLDMD5=`curl $URL | egrep "$DES" | md5sum | awk '{print $1}'`
sleep 300
fi
done
</code></p>
squid自动配置web化
2010-01-23T00:00:00+08:00
CDN
squid
web
bash
http://chenlinux.com/2010/01/23/automate-configure-squid-in-website
<p>[root@test data]# cat index.htm<br />
```html</p>
<html>
<head>
<title>饶琛琳专用21V-CDN流程系统</title>
<style type="text/css">
.style1 {
text-align: center;
}
</style>
</head>
<body>
<center>
<h1>Designed for support@21vianet.com in RT table.</h1>
<hr />
<form method="post" action="/cgi-bin/rt.cgi">
<table>
<tr>
<td colspan="2">
<textarea name="config" style="width: 700px; height: 200px" rows="1" cols="20" onmouseover="focus()" onfocus="if(value=='在此填入squid配置,或者者测试url,注意分行哟') {value=''}" onmouseout="blur()" onblur="if (value=='') {value='在此填入squid配置,或者测试url,注意分行哟'}">在此填入squid配置,或者测试url,注意分行哟</textarea></td>
</tr>
<tr>
<td>
<textarea name="iplist" style="width: 200px; height: 250px" rows="1" cols="20" onmouseover="focus()" onfocus="if(value=='在此填入服务器组IP,注意绵阳、广州、汕头网通没VPN哈') {value=''}" onmouseout="blur()" onblur="if (value=='') {value='在此填入服务器组IP,注意绵阳、广州、汕头网通没VPN哈'}">在此入服务器组IP,注意绵阳、广州、汕头网通没VPN哈</textarea></td>
<td>
<input name="custom" type="text" style="width: 246px" value="在此填入客户简称" onfocus="if(value=='在此填入客户简称') {value=''}" onblur="if (value=='') {value='在此填入客户简称'}" />
<br />
<input name="action" type="radio" value="add" />添加该客户加速配置<br />
<input name="action" type="radio" value="del" />删除该客户加速配置<br />
<input name="action" type="radio" value="change" />更新该客户加速配置<br />
<input name="action" type="radio" value="conf" />检查配置统一性<br />
<input name="action" type="radio" value="wget" />wget方式检查(https等特殊情况推荐)<br />
<input name="action" type="radio" value="curl" checked="checked" />curl方式检查(防盗链、过期时间推荐)<br />
<input name="submit" type="submit" /></td>
</tr>
</table>
</center>
<div id="ShowAD" style="position:absolute;z-index:100;font-size:12px">
<div style="width:135;height:18px;font-size:14px;font-weight:bold;text-align:left;cursor:hand;" onclick="closead();"><font color="ff0000">升级特性</font>
v0.1.0:配置文件存入中心,调用脚本conf.sh运行;<br />
v0.1.1:新增wget测试功能,采用hosts绑定方式;<br />
v0.2.0:新增curl测试、配置文件校对功能,采用指定代理参数,避免修改hosts权限问题;<br />
v0.3.0:更新配置分发功能,直接逐句插入;<br />
v0.3.1:加强wget和curl测试定制功能,支持HTTPS、防盗链、过期时间检查;<br />
v0.3.2:支持url批量测试;<br />
v0.3.3:加强输入IP自动识别;<br />
更多精彩,敬请期待~~<br />
<script language="javascript">
var bodyfrm = ( document.compatMode.toLowerCase()=="css1compat" ) ? document.documentElement : document.body;
var adst = document.getElementById("ShowAD").style;
adst.top = ( bodyfrm.clientHeight -300-22 ) + "px";
adst.left = ( bodyfrm.clientWidth -200 ) + "px";
function moveR() {
adst.top = ( bodyfrm.scrollTop + bodyfrm.clientHeight - 300-22) + "px";
adst.left = ( bodyfrm.scrollLeft + bodyfrm.clientWidth - 200 ) + "px";
}
setInterval("moveR();", 80);
function closead()
{
adst.display='none';
}
</script>
```
[root@BeiJingBGP-Dns-02 cgi-bin]# cat rt.cgi
```bash
#!/bin/bash
function filter(){
sed '{
s/%23/#/g;
s/%0D%0A/n/g;
s/%5E/^/g;
s/%3A/:/g;
s/%2F///g;
s/%28/\(/g;
s/%7C/\|/g;
s/%29/\)/g;
s/%24/$/g;
s/%25/%/g;
s/%3F/?/g;
s/%3D/=/g;
s/%5C/\/g;
}' $1
}
function regulate(){
sed "{
1i#$custom
$a#$custom+end
$!N; /^(.*)n1$/!P;D
}" $1
}
function uniform(){
awk -F"[+| ]" '{
for(i=1;i<=NF;i++){
if($i~/[0-9]+.[0-9]+.[0-9]+.[0-9]+/){
print $i
}
}
}'
}
echo "Content-type:text/html"
echo ""
echo "<html>"
if [ "$REQUEST_METHOD" = "POST" ] ; then
QUERY_STRING=`cat -`
fi
#echo "<center>$QUERY_STRING</center>"
action=`echo $QUERY_STRING|awk -F"[=|&]" '{print $8}'`
custom=`echo $QUERY_STRING|awk -F"[=|&]" '{print $6}'`
iplist=`echo $QUERY_STRING|awk -F"[=|&]" '{print $4}'|filter|uniform`
case $action in
add)
ref_conf=`echo $QUERY_STRING|awk -F"[=|&]" '{print $2}'|filter|regulate|sed '1!G;h;$!d'`
#echo "$ref_conf"
for ip in $iplist;do
ping -c 3 $ip|awk -F, '/loss/{print $3}'
for ref in $ref_conf;do
echo "$ref<br />"
/usr/local/bin/sshpass -p 123456 ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no <a href="mailto:root@$ip">root@$ip</a> sed -i "/config/a\`echo $ref|sed 's/+/\ /g'`" /home/squid/etc/squid.conf && /home/squid/sbin/squid -k reconfigure
done
done
;;
wget)
test_url=`echo $QUERY_STRING|awk -F"[=|&]" '{print $2}'|filter`
for ip in $iplist;do
for url in $test_url;do
domain=`echo $url|awk -F/ '{print $3}'`
echo "$ip $domain" > /etc/hosts
wget -S -O /dev/null "$url" -o wget.log --no-check-certificate -t 1
cat wget.log|awk 'BEGIN{ORS="<br />"}1'
done
done
;;
curl)
test_url=`echo $QUERY_STRING|awk -F"[=|&]" '{print $2}'|filter`
for url in $test_url;do
echo "##################################<br />"
echo "###$url<br />"
for ip in $iplist;do
domain=`echo $url|awk -F/ '{print $3}'`
echo "##############<br />"
echo "$ip<br />"
curl -I -x $ip:80 -A "support/RT (21V-CDN)" -e "<a href="http://$domain/">http://$domain</a>" "$url"|awk 'BEGIN{ORS="<br />"}/HTTP/1|Cache|Age/'
sleep 1;
done
done
;;
*)
echo '<head><meta http-equiv="refresh" content="0;URL=http://1.2.3.4/index.htm" /></head>'
;;
esac
echo "</html>"
```
</div></div></body></html>
系统优化——TCP参数
2010-01-19T00:00:00+08:00
linux
http://chenlinux.com/2010/01/19/sysctl-about-tcp
<p>tcp_syn_retries :INTEGER<br />
默认值是5<br />
对于一个新建连接,内核要发送多少个 SYN 连接请求才决定放弃。不应该大于255,默认值是5,对应于180秒左右时间。(对于大负载而物理通信良好的网络而言,这个值偏高,可修改为2.这个值仅仅是针对对外的连接,对进来的连接,是由tcp_retries1决定的)</p>
<p>tcp_synack_retries :INTEGER<br />
默认值是5<br />
对于远端的连接请求SYN,内核会发送SYN + ACK数据报,以确认收到上一个 SYN连接请求包。这是所谓的三次握手(threeway handshake)机制的第二个步骤。这里决定内核在放弃连接之前所送出的 SYN+ACK数目。不应该大于255,默认值是5 ,对应于180秒左右时间。(可以根据上面的 tcp_syn_retries来决定这个值)</p>
<p>tcp_keepalive_time :INTEGER<br />
默认值是7200(2小时)<br />
当keepalive打开的情况下,TCP发送keepalive消息的频率。(由于目前网络攻击等因素,造成了利用这个进行的攻击很频繁,曾经也有cu的朋友提到过,说如果2边建立了连接,然后不发送任何数据或者rst/fin消息,那么持续的时间是不是就是2小时,空连接攻击?tcp_keepalive_time就是预防此情形的.我个人在做nat服务的时候的修改值为 1800 秒)</p>
<p>tcp_keepalive_probes: INTEGER<br />
默认值是9<br />
TCP发送keepalive探测以确定该连接已经断开的次数。(注意:保持连接仅在SO_KEEPALIVE套接字选项被打开是才发送.次数默认不需要修改,当然根据情形也可以适当地缩短此值.设置为5比较合适)</p>
<p>tcp_keepalive_intvl :INTEGER<br />
默认值为75<br />
探测消息发送的频率,乘以tcp_keepalive_probes就得到对于从开始探测以来没有响应的连接杀除的时间。默认值为75秒,也就是没有活动的连接将在大约11分钟以后将被丢弃。(对于普通应用来说,这个值有一些偏大,可以根据需要改小.特别是web类服务器需要改小该值,15是个比较合适的值)</p>
<p>tcp_retries1 :INTEGER<br />
默认值是3<br />
放弃回应一个TCP连接请求前﹐需要进行多少次重试。RFC 规定最低的数值是3 ﹐这也是默认值﹐根据RTO的值大约在3秒 - 8分钟之间。(注意:这个值同时还决定进入的syn连接)<br />
tcp_retries2 :INTEGER<br />
默认值为15<br />
在丢弃激活(已建立通讯状况)的TCP连接之前﹐需要进行多少次重试。默认值为15<br />
,根据RTO的值来决定,相当于13-30分钟(RFC1122规定,必须大于100秒).(这个值根据目前的网络设置,可以适当地改小,我的网络内修改为了5)<br />
tcp_orphan_retries :INTEGER<br />
默认值是7<br />
在近端丢弃TCP连接之前﹐要进行多少次重试。默认值是7<br />
个﹐相当于 50秒 - 16分钟﹐视 RTO 而定。如果您的系统是负载很大的web服务器﹐那么也许需要降低该值﹐这类 sockets<br />
可能会耗费大量的资源。另外参的考 tcp_max_orphans 。(事实上做NAT的时候,降低该值也是好处显著的,我本人的网络环境中降低该值为3)<br />
tcp_fin_timeout :INTEGER<br />
默认值是 60<br />
对于本端断开的socket连接,TCP保持在FIN-WAIT-2状态的时间。对方可能会断开连接或一直不结束连接或不可预料的进程死亡。默认值为<br />
60 秒。过去在2.2版本的内核中是 180<br />
秒。您可以设置该值﹐但需要注意﹐如果您的机器为负载很重的web服务器﹐您可能要冒内存被大量无效数据报填满的风险﹐FIN-WAIT-2<br />
sockets 的危险性低于 FIN-WAIT-1 ﹐因为它们最多只吃 1.5K 的内存﹐但是它们存在时间更长。另外参考<br />
tcp_max_orphans 。(事实上做NAT的时候,降低该值也是好处显著的,我本人的网络环境中降低该值为30)<br />
tcp_max_tw_buckets :INTEGER<br />
默认值是180000<br />
系统在同时所处理的最大 timewait sockets<br />
数目。如果超过此数的话﹐time-wait socket 会被立即砍除并且显示警告信息。之所以要设定这个限制﹐纯粹为了抵御那些简单的<br />
DoS<br />
攻击﹐千万不要人为的降低这个限制﹐不过﹐如果网络条件需要比默认值更多﹐则可以提高它(或许还要增加内存)。(事实上做NAT的时候最好可以适当地增加该值)<br />
tcp_tw_recycle :BOOLEAN<br />
默认值是0<br />
打开快速<br />
TIME-WAIT sockets 回收。除非得到技术专家的建议或要求﹐请不要随意修改这个值。(做NAT的时候,建议打开它)<br />
tcp_tw_reuse :BOOLEAN<br />
默认值是0<br />
该文件表示是否允许重新应用处于TIME-WAIT状态的socket用于新的TCP连接(这个对快速重启动某些服务,而启动后提示端口已经被使用的情形非常有帮助)<br />
tcp_max_orphans :INTEGER<br />
缺省值是8192<br />
系统所能处理不属于任何进程的TCP<br />
sockets最大数量。假如超过这个数量﹐那么不属于任何进程的连接会被立即reset,并同时显示警告信息。之所以要设定这个限制﹐纯粹为了抵御那些简单的<br />
DoS 攻击﹐千万不要依赖这个或是人为的降低这个限制(这个值Redhat<br />
AS版本中设置为 32768<br />
,但是很多防火墙修改的时候,建议该值修改为<br />
2000 )<br />
tcp_abort_on_overflow :BOOLEAN<br />
缺省值是0<br />
当守护进程太忙而不能接受新的连接,就象对方发送reset消息,默认值是false。这意味着当溢出的原因是因为一个偶然的猝发,那么连接将恢复状态。只有在你确信守护进程真的不能完成连接请求时才打开该选项,该选项会影响客户的使用。(对待已经满载的sendmail,apache这类服务的时候,这个可以很快让客户端终止连接,可以给予服务程序处理已有连接的缓冲机会,所以很多防火墙上推荐打开它)<br />
tcp_syncookies :BOOLEAN<br />
默认值是 0<br />
只有在内核编译时选择了CONFIG_SYNCOOKIES时才会发生作用。当出现syn等候队列出现溢出时象对方发送syncookies。目的是为了防止syn<br />
flood攻击。<br />
注意:该选项千万不能用于那些没有收到攻击的高负载服务器,如果在日志中出现synflood消息,但是调查发现没有收到synflood攻击,而是合法用户的连接负载过高的原因,你应该调整其它参数来提高服务器性能。参考:<br />
tcp_max_syn_backlog<br />
tcp_synack_retries<br />
tcp_abort_on_overflow<br />
syncookie严重的违背TCP协议,不允许使用TCP扩展,可能对某些服务导致严重的性能影响(如SMTP转发)。(注意,该实现与BSD上面使用的tcp<br />
proxy一样,是违反了RFC中关于tcp连接的三次握手实现的,但是对于防御syn-flood的确很有用.)<br />
tcp_stdurg :BOOLEAN<br />
默认值为0<br />
使用 TCP urg pointer 字段中的主机请求解释功能。大部份的主机都使用老旧的 BSD解释,因此如果您在 Linux<br />
打开它﹐或会导致不能和它们正确沟通。<br />
tcp_max_syn_backlog :INTEGER<br />
对于那些依然还未获得客户端确认的连接请求﹐需要保存在队列中最大数目。对于超过 128Mb 内存的系统﹐默认值是<br />
1024 ﹐低于 128Mb 的则为 128 。如果服务器经常出现过载﹐可以尝试增加这个数字。警告﹗假如您将此值设为大于<br />
1024 ﹐最好修改<br />
include/net/tcp.h 里面的<br />
TCP_SYNQ_HSIZE ﹐以保持<br />
TCP_SYNQ_HSIZE*16<br />
﹐并且编进核心之内。(SYN<br />
Flood攻击利用TCP协议散布握手的缺陷,伪造虚假源IP地址发送大量TCP-SYN半打开连接到目标系统,最终导致目标系统Socket队列资源耗尽而无法接受新的连接。为了应付这种攻击,现代Unix系统中普遍采用多连接队列处理的方式来缓冲(而不是解决)这种攻击,是用一个基本队列处理正常的完全连接应用(Connect()和Accept()<br />
),是用另一个队列单独存放半打开连接。这种双队列处理方式和其他一些系统内核措施(例如Syn-Cookies/Caches)联合应用时,能够比较有效的缓解小规模的SYN<br />
Flood攻击(事实证明<br />
)<br />
tcp_window_scaling :INTEGER<br />
缺省值为1<br />
该文件表示设置tcp/ip会话的滑动窗口大小是否可变。参数值为布尔值,为1时表示可变,为0时表示不可变。tcp/ip通常使用的窗口最大可达到65535字节,对于高速网络,该值可能太小,这时候如果启用了该功能,可以使tcp/ip滑动窗口大小增大数个数量级,从而提高数据传输的能力(RFC1323)。(对普通地百M网络而言,关闭会降低开销,所以如果不是高速网络,可以考虑设置为0 )</p>
<p>tcp_timestamps :BOOLEAN<br />
缺省值为1<br />
Timestamps用在其它一些东西中﹐可以防范那些伪造的 sequence 号码。一条1G的宽带线路或许会重遇到带out-of-line数值的旧sequence 号码(假如它是由于上次产生的)。Timestamp 会让它知道这是个’旧封包’。(该文件表示是否启用以一种比超时重发更精确的方法(RFC1323)来启用对 RTT 的计算;为了实现更好的性能应该启用这个选项。)</p>
<p>tcp_sack :BOOLEAN<br />
缺省值为1<br />
使用 Selective ACK﹐它可以用来查找特定的遗失的数据报—因此有助于快速恢复状态。该文件表示是否启用有选择的应答(Selective Acknowledgment),这可以通过有选择地应答乱序接收到的报文来提高性能(这样可以让发送者只发送丢失的报文段)。(对于广域网通信来说这个选项应该启用,但是这会增加对CPU 的占用。)</p>
<p>tcp_fack :BOOLEAN<br />
缺省值为1<br />
打开FACK拥塞避免和快速重传功能。(注意,当tcp_sack 设置为0 的时候,这个值即使设置为1 也无效)</p>
<p>tcp_dsack :BOOLEAN<br />
缺省值为1<br />
允许TCP发送”两个完全相同”的SACK。</p>
<p>tcp_ecn :BOOLEAN<br />
缺省值为0<br />
打开TCP的直接拥塞通告功能。</p>
<p>tcp_reordering :INTEGER<br />
默认值是3<br />
TCP流中重排序的数据报最大数量 。 (一般有看到推荐把这个数值略微调整大一些,比如5 )</p>
<p>tcp_retrans_collapse :BOOLEAN<br />
缺省值为1<br />
对于某些有bug的打印机提供针对其bug的兼容性。(一般不需要这个支持,可以关闭它)</p>
<p>tcp_wmem (3个INTEGER 变量): min, default, max<br />
min :为TCP socket预留用于发送 缓冲 的内存最小值。每个tcp socket都可以在建议以后都可以使用它。默认值为4096(4K) 。<br />
default :为TCP socket预留用于发送缓冲的内存数量,默认情况下该值会影响其它协议使用的net.core.wmem_default值,一般要低于net.core.wmem_default 的值。默认值为16384(16K) 。<br />
max : 用于TCP socket发送缓冲的内存最大值。该值不会影响net.core.wmem_max,”静态”选择参数SO_SNDBUF则不受该值影响。默认值为131072(128K) 。(对于服务器而言,增加这个参数的值对于发送数据很有帮助,在我的网络环境中,修改为了51200 131072 204800)</p>
<p>tcp_rmem (3个INTEGER 变量): min , default , max</p>
<div class="highlighter-rouge"><pre class="highlight"><code>min :为TCP socket预留用于接收缓冲 的内存数量,即使在内存出现紧张情况下tcp socket都至少会有这么多数量的内存用于接收缓冲,默认值为8K 。
default :为TCP socket预留用于接收缓冲的内存数量,默认情况下该值影响其它协议使用的net.core.wmem_default值。该值决定了在tcp_adv_win_scale、tcp_app_win 和tcp_app_win=0默认值情况下,TCP窗口大小为65535。默认值为87380
max :用于TCP socket接收缓冲的内存最大值。该值不会影响net.core.wmem_max,"静态"选择参数 SO_SNDBUF则不受该值影响。默认值为 128K 。默认值为87380*2 bytes。(可以看出,.max的设置最好是default的两倍,对于NAT来说主要该增加它,我的网络里为51200 131072 204800)
</code></pre>
</div>
<p>tcp_mem (3个INTEGER 变量):low , pressure , high<br />
low :当TCP使用了低于该值的内存页面数 时,TCP不会考虑释放内存。(理想情况下,这个值应与指定给 tcp_wmem 的第 2 个值相匹配 - 这第 2 个值表明,最大页面大小乘以最大并发请求数除以页大小 (131072 * 300 / 4096 )。 )<br />
pressure :当TCP使用了超过该值的内存页面数量时,TCP试图稳定其内存使用,进入pressure模式,当内存消耗低于low值时则退出pressure状态。(理想情况下这个值应该是TCP 可以使用的总缓冲区大小的最大值 (204800 * 300 / 4096 )。 )<br />
high :允许所有tcp sockets用于排队缓冲数据报的页面量。(如果超过这个值,TCP 连接将被拒绝,这就是为什么不要令其过于保守 (512000 * 300 / 4096 ) 的原因了。</p>
<p>在这种情况下,提供的价值很大,它能处理很多连接,是所预期的 2.5 倍;或者使现有连接能够传输 2.5 倍的数据。我的网络里为192000 300000 732000)一般情况下这些值是在系统启动时根据系统内存数量计算得到的。</p>
<p>tcp_app_win : INTEGER<br />
默认值是31<br />
保留max(window/2^tcp_app_win, mss)数量的窗口由于应用缓冲。当为0时表示不需要缓冲。</p>
<p>tcp_adv_win_scale : INTEGER<br />
默认值为2<br />
计算缓冲开销bytes/2^tcp_adv_win_scale(如果tcp_adv_win_scale > 0)或者bytes-bytes/2^(-tcp_adv_win_scale)(如果tcp_adv_win_scale <= 0)。</p>
<p>tcp_rfc1337 :BOOLEAN<br />
缺省值为0<br />
这个开关可以启动对于在RFC1337中描述的”tcp的time-wait暗杀危机”问题的修复。启用后,内核将丢弃那些发往time-wait状态TCP套接字的RST包.</p>
<p>tcp_low_latency: BOOLEAN<br />
缺省值为0<br />
允许 TCP/IP 栈适应在高吞吐量情况下低延时的情况;这个选项一般情形是的禁用。(但在构建Beowulf集群的时候,打开它很有帮助)</p>
<p>tcp_westwood :BOOLEAN<br />
缺省值为0<br />
启用发送者端的拥塞控制算法,它可以维护对吞吐量的评估,并试图对带宽的整体利用情况进行优化;对于 WAN通信来说应该启用这个选项。</p>
<p>tcp_bic :BOOLEAN<br />
缺省值为0<br />
为快速长距离网络启用 Binary Increase Congestion;这样可以更好地利用以 GB 速度进行操作的链接;对于WAN 通信应该启用这个选项。</p>
squid刷新缓存
2010-01-13T00:00:00+08:00
CDN
squid
http://chenlinux.com/2010/01/13/refresh-cache-of-interrogation-url
<p>这篇博文的起因是最近犯的一个低级错误。某客户要求立刻刷新掉一批url。格式是http://a.b.com/index.jsp?pid=123456&id=654321这样的。<br />
我也没多想,上去就执行“squidclient -p 80 -m purge<br />
http://a.b.com/index.jsp?pid=123456&id=654321”。结果硬是刷不掉……明明访问源站已经没有这个url了。我wget居然还能MISS/200~~~<br />
直到同事提醒。才赫然发现其实我一直在purge和wget的,不是http://a.b.com/index.jsp?pid=123456&id=654321,而是http://a.b.com/index.jsp?pid=123456。linux把url里的&当作后台运行的命令执行了……而access.log中为了安全又没记录?后的具体参数,于是傻傻的跟客户客服扯皮了N久……<br />
“squidclient -p 80 -m purge ‘http://a.b.com/index.jsp?pid=123456&amp;id=654321’”这么一刷立刻就好了~~<br />
由此告诫自己,以后一定要严谨行事,引号最好还是要养成习惯都给带上~~</p>
<p>以下具体总结摘录squid缓存刷新的几种办法,随便感谢百度,悼念谷歌:</p>
<p>第一种——当squid的refresh_pattern未使用任何options,即squid缓存遵循http协议精神时有效。原理是模拟no-cache头的request(即ctrl+F5刷新):<br />
【nnd,一帖就错误,单独放一个帖子,大家看下一贴吧】</p>
<p>第二种,采用PURGE方式刷新,request的header如下:</p>
<p>PURGE http://www.lrrr.org/junk HTTP/1.0<br />
Accept: <em>/</em></p>
<p>脚本类似第一种方法,修改其中的HEAD为PURGE即可:</p>
<p>$head = “PURGE $url_component[‘path’] HTTP/1.1\r\n”;</p>
<p>按照《squid中文权威指南》的说法,当squid收到purge指令的时候,也是采用head和get的方式去处理请求,然后找到cache的文件进行删除的。</p>
<p>第三种,采用多播HTCP包。这是 MediaWiki 目前正在使用的方法,当wiki 更新时用于更新全球的 Squid缓存服务器,实现原理为:发送 PURGE 请求到特定的多播组,所有Squid服务器通过订阅该多播组信息完成删除操作,这种实现方式非常高效,避免了 Squid 服务器处理响应和建立 TCP连接的开销。参考资料: Multicast HTCP purging。</p>
<p>第五种,小工具:<a href="http://www.wa.apana.org.au/~dean/squidpurge/">http://www.wa.apana.org.au/~dean/squidpurge/</a><br />
wget http://www.wa.apana.org.au/~dean/sources/purge-20040201-src.tar.gz<br />
tar zxvf purge-20040201-src.tar.gz<br />
cd purge<br />
make<br />
./purge -help<br />
### Use at your own risk! No guarantees whatsoever. You were warned. ###<br />
$Id: purge.cc,v 1.17 2000/09/21 10:59:53 cached Exp $<br />
Usage: purge [-a] [-c cf] [-d l] [-(f|F) fn | -(e|E) re] [-ph[:p]]<br />
[-P #] [-s] [-v] [-C dir [-H]] [-n]<br />
-a display a little rotating thingy to indicate that I am alive<br />
(tty only).<br />
-c c squid.conf location, default<br />
“/usr/local/squid/etc/squid.conf”.<br />
-C dir base directory for content extraction (copy-out mode).<br />
-d l debug level, an or of different debug options.<br />
-e re single regular expression_r_r per -e instance (use<br />
quotes!).<br />
-E re single case sensitive regular expression_r_r like -e.<br />
-f fn name of textfile containing one regular<br />
expression_r_r per line.<br />
-F fn name of textfile like -f containing case sensitive REs.<br />
-H prepend HTTP reply header to destination files in copy-out<br />
mode.<br />
-n do not fork() when using more than one cache_dir.<br />
-p h:p cache runs on host h and optional port p, default is<br />
localhost:3128.<br />
-P # if 0, just print matches; otherwise or the following purge<br />
modes:<br />
0x01 really send PURGE to the cache.<br />
0x02 remove all caches files reported as 404 (not found).<br />
0x04 remove all weird (inaccessible or too small) cache<br />
files.<br />
0 and 1 are recommended - slow rebuild your cache with other modes.<br />
-s show all options after option parsing, but before really<br />
starting.<br />
-v show more information about the file, e.g. MD5, timestamps and<br />
flags.</p>
<p>使用示例:</p>
<ol>
<li>清除URL中包含jackbillow.com的所有缓存<br />
./purge -p 127.0.0.1:80 -P 1 -se ‘jackbillow.com’</li>
<li>清除 URL 以“.mp3”结尾的缓存文件,例如:http://www.dzend.com/abc/test.mp3<br />
./purge -p 127.0.0.1:80 -P 1 -se ‘.mp3$’</li>
</ol>
<p>第五种,张宴的脚本clear_squid_cache.sh。(老小注:还是这个熟悉,呵呵)<br />
<code class="highlighter-rouge">bash
#!/bin/sh
squidcache_path="/data1/squid/var/cache"
squidclient_path="/usr/local/squid/bin/squidclient"
grep -a -r $1 $squidcache_path/* | strings | grep "http:" | awk -F'http:' '{print "http:"$2;}' >cache_list.txt
for url in `cat cache_list.txt`; do
$squidclient_path -m PURGE -p 80 $url
done
</code></p>
<p>据说:经测试,在DELL 2950上清除26000个缓存文件用时2分钟左右。平均每秒可清除缓存文件177个。<br />
看到了吧,网上流传的这个脚本里,$url也没有用”“引起来,所以用这个sh刷新的时候,也失败了……</p>
刷新squid缓存的php脚本
2010-01-13T00:00:00+08:00
CDN
php
squid
http://chenlinux.com/2010/01/13/refresh-cache-by-php
<div class="highlighter-rouge"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="k">interface</span> <span class="nx">Flush_Cache</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">flush</span><span class="p">(</span><span class="nv">$url</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">class</span> <span class="nc">Flush_Cache_HTTP_Header_Impl</span> <span class="k">implements</span> <span class="nx">Flush_Cache</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">flush</span><span class="p">(</span><span class="nv">$url</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="k">empty</span><span class="p">(</span><span class="nv">$url</span><span class="p">))</span>
<span class="p">{</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">$url_component</span> <span class="o">=</span> <span class="nb">parse_url</span><span class="p">(</span><span class="nv">$url</span><span class="p">);</span>
<span class="k">global</span> <span class="nv">$g_squid_servers</span><span class="p">;</span>
<span class="k">foreach</span> <span class="p">(</span><span class="nv">$g_squid_servers</span> <span class="k">as</span> <span class="nv">$server</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$squid_params</span> <span class="o">=</span> <span class="nb">split</span><span class="p">(</span><span class="s1">':'</span> <span class="p">,</span> <span class="nv">$server</span><span class="p">);</span>
<span class="nv">$fsocket</span> <span class="o">=</span> <span class="nb">fsockopen</span><span class="p">(</span><span class="nv">$squid_params</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="nb">intval</span><span class="p">(</span><span class="nv">$squid_params</span><span class="p">[</span><span class="mi">1</span><span class="p">]),</span> <span class="nv">$errono</span><span class="p">,</span> <span class="nv">$errstr</span><span class="p">,</span> <span class="mi">3</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="k">FALSE</span> <span class="o">!=</span> <span class="nv">$fsocket</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$head</span> <span class="o">=</span> <span class="s2">"HEAD </span><span class="si">{</span><span class="nv">$url_component</span><span class="p">[</span><span class="s1">'path'</span><span class="p">]}</span> <span class="nx">HTTP</span><span class="o">/</span><span class="mf">1.1</span><span class="nx">rn</span><span class="s2">";
</span><span class="nv">$head</span><span class="s2"> .= "</span><span class="nx">Accept</span><span class="o">:</span> <span class="o">*/*</span><span class="nx">rn</span><span class="s2">";
</span><span class="nv">$head</span><span class="s2"> .= "</span><span class="nx">Host</span><span class="o">:</span> <span class="p">{</span><span class="nv">$url_component</span><span class="p">[</span><span class="s1">'host'</span><span class="p">]}</span><span class="nx">rn</span><span class="s2">";
</span><span class="nv">$head</span><span class="s2"> .= "</span><span class="nx">Cache</span><span class="o">-</span><span class="nx">Control</span><span class="o">:</span> <span class="nx">no</span><span class="o">-</span><span class="nx">cachern</span><span class="s2">";
</span><span class="nv">$head</span><span class="s2"> .= "</span><span class="nx">rn</span><span class="s2">";
echo </span><span class="nv">$head</span><span class="s2">;
fwrite(</span><span class="nv">$fsocket</span><span class="s2"> , </span><span class="nv">$head</span><span class="s2">);
while (!feof(</span><span class="nv">$fsocket</span><span class="s2">))
{
</span><span class="nv">$line</span><span class="s2"> = fread(</span><span class="nv">$fsocket</span><span class="s2"> , 4096);
echo </span><span class="nv">$line</span><span class="s2">;
}
fclose(</span><span class="nv">$fsocket</span><span class="s2">);
}
}
}
}
</span><span class="nv">$g_squid_servers</span><span class="s2"> = array('192.168.2.88:80');
</span><span class="nv">$flush_cache</span><span class="s2"> = new Flush_Cache_HTTP_Header_Impl();
</span><span class="nv">$flush_cache->flush</span><span class="s2">('http://ent.cdqss.com/index.html');
?>
</span></code></pre>
</div>
Squid的ACL分类
2010-01-10T00:00:00+08:00
squid
http://chenlinux.com/2010/01/10/zz-intro-acls-in-squid
<p>在找http_status的acl用法时,看到这么一句“This clause supports both fast and slow acl types”。acl还分快慢呢~~赶紧去wiki看:<a href="http://wiki.squid-cache.org/SquidFaq/SquidAcl#Fast_and_Slow_ACLs">http://wiki.squid-cache.org/SquidFaq/SquidAcl#Fast_and_Slow_ACLs</a></p>
<div class="highlighter-rouge"><pre class="highlight"><code>Some ACL types require information which may not
be already available to Squid. Checking them requires suspending
work on the current request, querying some external source, and
resuming work when the needed information becomes available. This
is for example the case for DNS, authenticators or external
authorization scripts. ACLs can thus be divided in
FAST ACLs, which do not require going to external
sources to be fulfilled, and SLOW ACLs, which do.
Fast ACLs include (as of squid 3.1.0.7):
all (built-in)
src
dstdomain
dstdom_regex
myip
arp
src_as
peername
time
url_regex
urlpath_regex
port
myport
myportname
proto
method
http_status {R}
browser
referer_regex
snmp_community
maxconn
max_user_ip
req_mime_type
req_header
rep_mime_type {R}
user_cert
ca_cert
Slow ACLs include:
dst
dst_as
srcdomain
srcdom_regex
ident
ident_regex
proxy_auth
proxy_auth_regex
external
ext_user
ext_user_regex
This list may be incomplete or out-of-date. See
your squid.conf.documented file for details. ACL types
marked with {R} are reply ACLs, see the dedicated FAQ
chapter.
Squid caches the results of ACL lookups whenever
possible, thus slow ACLs will not always need to go to the external
data-source.
Knowing the behaviour of an ACL type is relevant
because not all ACL matching directives support all kinds of ACLs.
Some check-points will not suspend the request:
they allow (or deny) immediately. If a SLOW acl has to be checked,
and the results of the check are not cached, the corresponding ACL
result will be as if it didn't match. In other words, such ACL
types are in general not reliable in all access check clauses.
</code></pre>
</div>
client-cache-origin之间的session问题
2010-01-09T00:00:00+08:00
CDN
squid
http://chenlinux.com/2010/01/09/keep-sessions-between-client-cache-origin
<p>昨天,突然接到某客户的邮件,表示他们质疑我们cache对其多originserver的轮询不均等,以至于影响到网站的访问。<br />
听起来不是什么难题,把服务器上的DNSCache进程中止掉,不就能平均轮询了么?可是操作完成后,客户依然不认可……于是开始细细探讨具体的错误问题所在。<br />
原来实际情况是这样:点击其网站的回复、收藏等动态页面时,时常会弹出错误页面。对这个错误页面,客户的解释是“session状态CDN加速未保留”。<br />
由此解释,我觉得cache上的问题恰恰相反,正是因为均等轮询了导致的。而客户其他一些单源的域名服务正常,也证明了这点。<br />
上cache服务器查看配置,使用的默认keep-alive设置,也就是对client和origin都开启了keep-alive,默认时间为2min。但实际上并没有起作用。在测试中,可以看到这样的日志:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[rao@localhost ~]$ tail access.log|awk '{print $4,$6,$7,$9,$10}'
TCP_IMS_HIT/304 GET http://a.b.com/wentiyongpin/11106917.html NONE/- "http://a.b.com/wentiyongpin/"
TCP_MISS/200 GET http://a.b.com/RelatedUserInfo.aspx?nickname=shunfataiqiu DIRECT/1.2.3.4 "http://a.b.com/wentiyongpin/11106917.html"
TCP_MISS/200 GET http://a.b.com/postcomment/Reply.aspx?info_id=11106917 DIRECT/1.2.3.5 "http://a.b.com/wentiyongpin/11106917.html"
TCP_MISS/302 POST http://a.b.com/postcomment/Reply.aspx?info_id=11106917 DIRECT/1.2.3.4 "http://a.b.com/postcomment/Reply.aspx?info_id=11106917"
TCP_MISS/404 GET http://a.b.com/nf3.aspx?aspxerrorpath=/postcomment/Reply.aspx DIRECT/1.2.3.4 "http://a.b.com/postcomment/Reply.aspx?info_id=11106917"
</code></pre>
</div>
<p>由日志可见,在进入post页面(此时方式还是GET)时,session的origin从1.2.3.4轮询到了1.2.3.5,而填完了回复内容,点击提交(此时方式改成POST)时,origin又轮询回了1.2.3.4——而此时1.2.3.4上并没有相应ID的session存在——于是页面被302重定向去了错误提示页面,也就是下面的404。</p>
<p>这种情况,主要来说,还是客户网站本身的架构问题。简单点,只用一个origin;复杂点,在多台webserver与后台数据库之间建共享连接池。保证session调用正常。</p>
<p>但在客户修改origin之前,cache本身能不能作出一定的改变呢?能。放弃使用dns查询,而采用squid本身的peer功能,就能搞定它,配置如下:<br />
<code class="highlighter-rouge">squid
#Parent
acl ParentDomain dstdomain a.b.com
cache_peer 1.2.3.4 parent 80 0 no-query no-netdb-exchange originserver sourcehash
cache_peer 1.2.3.5 parent 80 0 no-query no-netdb-exchange originserver sourcehash
cache_peer_access 1.2.3.4 allow ParentDomain
cache_peer_access 1.2.3.5 allow ParentDomain
always_direct allow !ParentDomain
#Parent end
</code><br />
用的sourcehash参数,相同的clientIP,使用相同的originIP,多好的loadbalance呀,更巧的是这个option,正好是squid2.6.STABLE21能用的,连2.7都没有,哈哈~~reconfigure后的正常日志如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>[rao@localhost ~]$ tail access.log|awk '{print $4,$6,$7,$9,$10}'
TCP_IMS_HIT/304 GET http://a.b.com/wentiyongpin/11106917.html NONE/- "http://a.b.com/wentiyongpin/"
TCP_MISS/200 GET http://a.b.com/postcomment/Reply.aspx?info_id=11106917 SOURCEHASH_PARENT/1.2.3.4 "http://a.b.com/wentiyongpin/11106917.html"
TCP_MISS/200 GET http://a.b.com/RelatedUserInfo.aspx?nickname=shunfataiqiu SOURCEHASH_PARENT/1.2.3.4 "http://a.b.com/wentiyongpin/11106917.html"
TCP_MISS/200 POST http://a.b.com/postcomment/Reply.aspx?info_id=11106917 SOURCEHASH_PARENT/1.2.3.4 "http://a.b.com/postcomment/Reply.aspx?info_id=11106917"
TCP_MISS/200 GET http://a.b.com/ SOURCEHASH_PARENT/1.2.3.4 "-"
TCP_MISS/200 GET http://a.b.com/zufang/ SOURCEHASH_PARENT/1.2.3.4 "http://a.b.com/"
TCP_MISS/200 GET http://a.b.com/ad.ashx?ad=ad&url=http://a.b.com/zufang/&alias=zufang&childalias=zufang SOURCEHASH_PARENT/1.2.3.4 "http://a.b.com/zufang/"
</code></pre>
</div>
<p>整个访问中,回源IP一直都是1.2.3.4,而且POST不再是302转404,而是200了~~</p>
squid3新acl类型http_status试用(源站故障转向研究)
2010-01-09T00:00:00+08:00
squid
http://chenlinux.com/2010/01/09/intro-http_status
<p>今天试着自己编译安装了squid3.1,然后开始移植2.6的配置文件。在squid.conf.document(2.6里的squid.conf.default)里看了很久,发现不少新东西。跟转向有关的,便发现一个:</p>
<p>acl aclname http_status 200 301 500- 400-403<br />
…<br />
# status code in reply</p>
<p>这个acl配合http_access和deny_info,也是个转向的办法。(相反的,大家举例时一般用来deny的正是302跳转)</p>
<p>acl err http_status 500-<br />
http_access deny err<br />
deny_info http://err.tiaozhuan.com err</p>
<p>不过问题依旧:这个依然没法把不同的域名分开——deny_info只针对自己上头那个http_access deny的aclname,可要是写上同样aclname的astdomain或者url_regex,整个访问就都废了……<br />
然后想到url_rewrite_access,如果用acl http_status配合url_rewrite_access,能不能做到避免所有请求回源确认呢?<br />
按照之前记录的squid处理流程,squid应该是在和client建立连接后的第二步就检查acl。但3.1中添加的这个http_status,总不可能在命中的第四步或者回源的第六步之前,就能完成访问控制呀?<br />
看来squid的内部机制,已经有了较大变化。<br />
明天做个实验,看看实际到底如何吧~</p>
<hr />
<p>经过试验,第一个想法不可行。因为deny_info的status是TCP_DENY/302,这和acl是冲突的,导致无法工作。从此也能看出,http_status的acl确实是在比较晚的流程中才起作用的。分别使用http_status和url_regex做deny的日志如下:<br />
1263112949.310 27099 12.34.56.78 TCP_MISS/502 1878 GET http://a.b.com/duanzu/ - DIRECT/1.2.3.4 text/html “http://a.b.com/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQPinyinSetup 620; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)”<br />
1263113026.271 0 12.34.56.78 TCP_DENIED/302 331 GET http://a.b.com/duanzu/ - NONE/- text/html “http://a.b.com/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; QQPinyinSetup 620; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)”</p>
<p>acl err http_status 500-<br />
url_rewrite_access deny !err</p>
<p>不起作用,所有的请求都不经过rewrite直接输出了……<br />
郁闷,这个新acl的用法还得慢慢找资料~~</p>
虚拟机时间问题的机制原理及解决办法
2010-01-05T00:00:00+08:00
cloud
xen
http://chenlinux.com/2010/01/05/principle-and-solution-of-the-vm-date-problem
<p>对于CDN来说,时间是个非常重要的概念,squid的缓存刷新控制,全都是以服务器时间为基准。在我博客的前不知道几篇的某文中,就提到过因为时间的原因引起的刷新事故。而最新的一次,是公司在日志流量统计时,因为相差太大导致截取失败——这可都是钱呢。<br />
实际工作中,大多会搭建一个NTP服务器,给应用服务器提供时间同步服务。但虚拟机在这方面有个问题。虚拟机的时间是读取的宿主机的硬件时钟,随便你date<br />
-s 1234怎么改,下一次date查看,都会恢复成原样。<br />
这样看起来是个好事,因为这样可以省略掉大批机器的NTP服务器同步时间的压力。但糟糕的事情在后头:一大批重要客户的虚拟机,在运行上半年以上时间后,date都比真实时间超前了(偶然我也看到过一次滞后的)。或许是几分钟,或许是几小时!!<br />
这一度让我很无奈,因为即使我重启虚拟机,这个诡异的date也依然倔强的按照它自己的时钟前进。看起来办法只有一个:把虚拟平台整个的重启——好在这种试验的结果是有效的,不然真不知道怎么办了。<br />
从这个无奈的办法中,也应该可以推测出一个结论,虚拟机的时钟,应该是另有文件存储在虚拟平台的某个临时文件里。所以关闭虚拟机,该文件还存在并继续计时;重启动虚拟平台,该文件才会重计。至于这个文件叫什么,存放在哪里,一时半会我就找不到了,或许这个问题不算什么特别大的问题,至少我在linuxsir.org的《xen初学指南》中,没有找到关于时钟的配置说明。<br />
不过再小的问题也顶不住人的研究。于是我找到了另一篇很对口的文章:《虚拟机中GUEST OS时钟(TIMEKEEP)问题的探讨》。<br />
文章告诉我们:虚拟机的时钟,与真实机的时钟区别,其一在于其处理不是抢占中断而是延迟处理;其二在于其中断模拟软件可能因资源占用问题被推迟。<br />
这两点区别,导致的结果却是一个,就是一旦服务器负载变高,时钟中断就被排到后头去了……<br />
文章也提出了解决办法,一个是编程,让系统补上中断的时间,这个俺就不说了;一个是修改内核,让系统不断修正date;<br />
进一步问题:不管哪个办法,第一,加重了虚拟机负载;第二,这又依赖与系统去更新date的进程的精准度,而这个进程的中断时钟如果有偏差的话,……<br />
再进一步方法……有兴趣的还是自己点文章看吧。<br />
看起来这个问题就要永远终结在此了。<br />
不过变通的主意及时的出现了——既然虚拟机跟虚拟平台之间的异步时钟同步没法解决,我们就干脆不要虚拟机跟虚拟平台同步了。这样虽然对NTP的压力大了,但毕竟保证了CDN的正常。而且从上头的原理可以知道,能够导致date偏差到这个程度的原因,大概也是因为虚拟机的负载压力太大。事实上,我们为了保证冗余,也是尽量避免这种压力出现的。那么date偏差的服务器,其实也就是少数了。<br />
顺着这个思路一查,原来办法其实很简单——修改虚拟机的independent_wallclock值即可:<br />
echo 1 > /proc/sys/xen/independent_wallclock<br />
或sysctl xen.independent_wallclock=1<br />
启用虚拟机独立系统时间;<br />
/etc/init.d/ntpd stop<br />
ntpdate 192.168.0.1<br />
就能完成NTP时间同步了;<br />
再date看看,时间果然就更正了~~<br />
/proc下的文件都是进程使用中的文件,如果要让虚拟机开机自动读取配置,把这个修改写进/etc/sysctl.conf才行哦:<br />
echo “xen.independent_wallclock = 1” » /etc/sysctl.conf<br />
就这么简单~~~</p>
squid限速
2010-01-01T00:00:00+08:00
squid
http://chenlinux.com/2010/01/01/limit-speed-in-squid
<p>squid有个delay_pool,可以做限速,虽然效果不太准~(嗯,就像限制并发连接数的maxconn一样)<br />
首先搬个老虎皮做大旗——《Squid: The Definitive Guide》的相关段落:</p>
<p>The buckets don’t actually store bandwidth (e.g., 100 Kbit/s), but <br />
rather some amount of traffic (e.g., 384 KB). Squid adds some <br />
amount of traffic to the buckets each second. Cache clients take <br />
some amount of traffic out when they receive data from an upstream <br />
source (origin server or neighbor). <br />
The size of a bucket determines how much burst bandwidth is <br />
available to a client. If a bucket starts out full, a client can <br />
take as much traffic as it needs until the bucket becomes empty. <br />
The client then receives traffic allotments at the fill rate. <br />
The mapping between Squid clients and actual buckets is a bit <br />
complicated. Squid uses three different constructs to do it: access <br />
rules, delay pool classes, and types of buckets. First, Squid <br />
checks a client request against the delay_access list. If the <br />
request is a match, it points to a particular delay pool. Each <br />
delay pool has a class: 1, 2, or 3. The classes determine which <br />
types of buckets are in use. Squid has three types of buckets: <br />
aggregate, individual, and network: <br />
A class 1 pool has a single aggregate bucket. <br />
A class 2 pool has an aggregate bucket and 256 individual <br />
buckets. <br />
A class 3 pool has an aggregate bucket, 256 network buckets, and <br />
65,536 individual buckets. <br />
As you can probably guess, the individual and network buckets <br />
correspond to IP address octets. In a class 2 pool, the individual <br />
bucket is determined by the last octet of the client’s IPv4 <br />
address. In a class 3 pool, the network bucket is determined by the <br />
third octet, and the individual bucket by the third and fourth <br />
octets. <br />
For the class 2 and 3 delay pools, you can disable buckets you <br />
don’t want to use. For example, you can define a class 2 pool with <br />
only individual buckets by disabling the aggregate bucket. <br />
When a request goes through a pool with more than one bucket type, <br />
it takes bandwidth from all buckets. For example, consider a class <br />
3 pool with aggregate, network, and individual buckets. If the <br />
individual bucket has 20 KB, the network bucket 30 KB, but the <br />
aggregate bucket only 2 KB, the client receives only a 2-KB <br />
allotment. Even though some buckets have plenty of traffic, the <br />
client is limited by the bucket with the smallest amount. <br />
C.2 Configuring Squid <br />
Before you can use delay pools, you must enable the feature when <br />
compiling. Use the —enable-delay-pools option when running <br />
./configure. You can then use the following directives to set up <br />
the delay pools. <br />
C.2.1 delay_pools <br />
The delay_pools directive tells Squid how many pools you want to <br />
define. It should go before any other delay pool-configuration <br />
directives in squid.conf. For example, if you want to have five <br />
delay pools: <br />
delay_pools 5 <br />
The next two directives actually define each pool’s class and other <br />
characteristics. <br />
C.2.2 delay_class <br />
You must use this directive to define the class for each pool. For <br />
example, if the first pool is class 3: <br />
delay_class 1 3 <br />
Similarly, if the fourth pool is class 2: <br />
delay_class 4 2 <br />
In theory, you should have one delay_class line for each pool. <br />
However, if you skip or omit a particular pool, Squid doesn’t <br />
complain. <br />
C.2.3 delay_parameters <br />
Finally, this is where you define the interesting delay pool <br />
parameters. For each pool, you must tell Squid the fill rate and <br />
maximum size for each type of bucket. The syntax is: <br />
delay_parameters N rate/size [rate/size [rate/size]] <br />
The rate value is given in bytes per second, and size in total <br />
bytes. If you think of rate in terms of bits per second, you must <br />
remember to divide by 8. <br />
Note that if you divide the size by the rate, you’ll know how long <br />
it takes (number of seconds) the bucket to go from empty to full <br />
when there are no clients using it. <br />
A class 1 pool has just one bucket and might look like this: <br />
delay_class 2 1 <br />
delay_parameters 2 2000/8000 <br />
For a class 2 pool, the first bucket is the aggregate, and the <br />
second is the group of individual buckets. For example: <br />
delay_class 4 2 <br />
delay_parameters 4 7000/15000 3000/4000 <br />
Similarly, for a class 3 pool, the aggregate bucket is first, the <br />
network buckets are second, and the individual buckets are <br />
third: <br />
delay_class 1 3 <br />
delay_parameters 1 7000/15000 3000/4000 1000/2000 <br />
C.2.4 delay_initial_bucket_level <br />
This directive sets the initial level for all buckets when Squid <br />
first starts or is reconfigured. It also applies to individual and <br />
network <br />
buckets, which aren’t created until first referenced. The value is <br />
a percentage. For example: <br />
delay_initial_bucket_level 75% <br />
In this case, each newly created bucket is initially filled to 75% <br />
of its maximum size. <br />
C.2.5 delay_access <br />
This list of access rules determines which requests go through <br />
which delay pools. Requests that are allowed go through the delay <br />
pools, while those that are denied aren’t delayed at all. If you <br />
don’t have any delay_access rules, Squid doesn’t delay any <br />
requests. <br />
The syntax for delay_access is similar to the other access rule <br />
lists (see Section 6.2), except that you must put a pool number <br />
before the allow or deny keyword. For example: <br />
delay_access 1 allow TheseUsers <br />
delay_access 2 allow OtherUsers <br />
Internally, Squid stores a separate access rule list for each delay <br />
pool. If a request is allowed by a pool’s rules, Squid uses that <br />
pool and stops searching. If a request is denied, however, Squid <br />
continues examining the rules for remaining pools. In other words, <br />
a deny rule causes Squid to stop searching the rules for a single <br />
pool but not for all pools. <br />
C.2.6 cache_peer no-delay Option <br />
The cache_peer directive has a no-delay option. If set, it makes <br />
Squid bypass the delay pools for any requests sent to that <br />
neighbor.</p>
<p>然后说老实话:我也看不太懂……<br />
只好贴一些百度出来的结果:<br />
class类型1为单个IP地址流量 <br />
class类型2为C类网段中的每个IP地址流量 <br />
class类型3为B类网段中的每个C类网段中的每个IP地址流量 <br />
具体的说:<br />
类型1只有一个总带宽流量实际也就是这个IP地址的流量 <br />
delay_parameters 1 64000/64000 <br />
类型2有两个带宽流量参数,第一个为整个C类型网段流量,第二个为每个IP流量 <br />
delay_parameters 1 -1/-1 64000/64000 <br />
类型3有三个带宽流量参数,第一个为整个B类网总流量,第二个为每个B类网段中的C类网段总流量,第三个为了B类网段中每个C类网段中的每个IP流量 <br />
delay_parameters 1 -1/-1 -1/-1 64000/64000 <br />
但似乎我还没百度到谁用class为2或者3的。一般大家都只用1……<br />
举个例子:<br />
两个域名,分别限制网民下载速度为50kb/s和100kb/s。配置如下:<br />
<code class="highlighter-rouge">squid
#定义域名
acl LIMIT_A dstdomain a.test.com
acl LIMIT_B dstdomain b.test.com
#定义受限IP段
acl LIMIT_IP src 192.168.1.0/24
acl ALL src 0/0
#开启两个连接延迟池
delay_pools 2
#定义两个延迟池,class类型均为1
delay_class 1 1
delay_class 2 1
#分配域名到不同的延迟池
delay_access 1 allow LIMIT_A
delay_access 2 allow LIMIT_B
#受限网段延迟池
delay_access 1 allow LIMIT_IP
#定义下载速率,速率定位为restore(bytes/sec)/max(bytes),,restore是表示以bytes/sec的速度下載object到bucket裡,而max則表示buckets的bytes值
delay_parameters 1 50000/50000
delay_parameters 2 100000/100000
#squid启动时初始化的池的带宽百分比
delay_initial_bucket_level 100
</code><br />
据网友的测试,当限速配置为20000/20000即20000/1024=19.53kb/s的时候,实际的下载速度大概在11-15kb/s之间。</p>
squid请求处理流程(源站故障转向研究)
2009-12-31T00:00:00+08:00
squid
http://chenlinux.com/2009/12/31/process-of-request-to-squid
<p>今天继续想别的办法,第一,当然还是修改源代码,这个C++可惜俺不懂,只是大略的通过百大哥谷大婶知道了squid的请求处理流程:</p>
<ol>
<li>客户端和squid建立连接(client-side模块、clientBeginRequest()函数);</li>
<li>检查ACL访问控制;</li>
<li>检查重定向;</li>
<li>检查缓存命中(GetMoreData()函数),写入StoreEntry(client-side模块);<br />
4.1. 命中(client-side模块);<br />
4.2. 未命中(rotoDispatch()函数启动peer算法,算法检查never|always_direct);</li>
<li>收到ICP响应,选择中止,转发请求(protoStart()函数);</li>
<li>打开到源站或peer的连接(HTTP模块),发起请求(NetworkCommunication模块),建立连接并处理异常(comm.c程序);</li>
<li>建立写缓存(HTTP模块),将请求写入socket;</li>
<li>建立相应的socket读缓存,接受处理HTTP响应(即,如果已有socket,可以跳过6、7步);</li>
<li>响应被接受,squid接收到header信息,并在被读取时把data追加进StoreEntry,同时通知client-side模块,这个过程的速度取决于delay_pools;</li>
<li>client-side模块从StoreEntry取数据,并写入客户端socket;</li>
<li>客户端读取完成,数据根据情况(refresh、cache)存入磁盘;</li>
<li>回源取完数据,标记StoreEntry为“完成”(client-side模块),socket关闭或保留到持久连接池;</li>
<li>数据写入客户端socket完成,从StoreEntry注释掉client-side模块,同样,关闭或等待客户端连接请求。</li>
</ol>
<p>原先的想法就是在第3步,而缓存在第4步,所以不行。也就是说,必须在第4步以后操作,才能不影响CDN的命中率。很显然,我注意到一个词,第6步中“建立链接并处理异常”的comm.c程序;其次,第9步“响应被接受”,也就是说,可能建立链接但没响应,这里就应该是socket的存活过期时间来决定。<br /></p>
<p>思路到此为止,往下就是编程开发能力的事儿了,哭ing~~</p>
<p>第二个办法,还在squid本身上做文章。我注意到,处理流程的4.2步上,squid是同时检查源站和cache_peer的。那么,在源站故障的时候,cache_peer可以设成其他服务器顶上。这样的情况,不单解决一个url跳转,还能做一个源站备份。配置如下:</p>
<pre><code class="language-squid">nonhierarchical_direct off
prefer_direct on
cache_peer 192.168.1.1 parent 80 0 no-query originserver name=S1
cache_peer 192.168.1.2 parent 80 0 no-query name=S2
cache_peer 192.168.1.3 parent 80 0 no-query name=S3
acl myservice dstdomain .site.com
cache_peer_access S1 allow myservice
cache_peer_access S2 allow myservice
cache_peer_access S3 allow myservice
</code></pre>
<p>第一句的意思,就是当originserver挂了,不可层叠的请求(即hierarchy_stoplist定义的)就“可能”发往其他peer。</p>
<p>第二句的意思,就是所有优先originserver,只有挂了以后才去peer,这个配置针对的可cache的请求。</p>
<p>不过两句加起来,还有那些不再hierarchy_stoplist定义又不cache的(没实验,不知道这个可cache是指squid配置还是包括origin的header设置),可就没办法了,也不全面。</p>
awk中让人郁闷的system()函数
2009-12-30T00:00:00+08:00
bash
awk
http://chenlinux.com/2009/12/30/system-function-in-awk
<p>发现一个特尴尬的事实。我辛辛苦苦去百度资料,想用rewrite实现针对不同域名源站故障后的自动跳转功能,但整个思路里遗漏了一个严重的问题。</p>
<p>按我的思路,针对请求的url进行一次curl,然后根据http_code去改写url或者原样输出——这也就意味着,每一个请求,squid都回源去取一次header。那么对于源站来说,前面squid的缓存率,就是0%!完全没有效果。</p>
<p>得重新想过办法……难道去看squid源代码?汗</p>
<p>本着有头有尾善始善终的原则,决定还是把原先那个鸡肋想法写完。根据squid权威指南11章的说法,传递给重定向器的流格式为:URL IP/FQDN IDENT METHOD,其中FQDN和ident经常是空。METHOD,一般是GET和POST,squid只能缓存GET的数据,但不能无视POST方式,因为有时候POST数据header太大的话,squid可能拒绝转发这些内容,这就不好玩了。</p>
<p>在明确这个格式以后(主要是草草收尾的想法影响下),我便觉得其实完全不用perl或者php来搞,简单的awk就足够了——当然,shell不行,因为shell不能从事这种流状的行处理。</p>
<p>以下是本着我想法写的awk脚本:<br />
<code class="highlighter-rouge">bash
#!/bin/awk -f
{
if(system("curl -o /dev/null -s -w %{http_code}" $1)~/^[2|3]/){
print ":$1"
} else {
print ":http://www.baidu.com/"
}
}
</code></p>
<p>但是再度让我郁闷的事情接连发生。</p>
<p>第一,不管我在{}中进行什么操作,程序都把system()的结果print出来了;</p>
<table>
<tbody>
<tr>
<td>第二,即使system()的结果是200,print出来的也是else{}的”http://www.baidu.com”;而如果我直接试验if(200~/^[2</td>
<td>3]/){}else{},结果就很正常!</td>
</tr>
</tbody>
</table>
<p>试验过程如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">[</span>rao@localhost ~]<span class="nv">$ </span><span class="nb">echo</span> <span class="s2">"http://www.google.com"</span>|awk <span class="s1">'{if(200~/^[2|3]/){ print ":"$1 } else{ print ":http://www.baidu.com/"}}'</span>
:http://www.google.com
<span class="o">[</span>rao@localhost ~]<span class="nv">$ </span><span class="nb">echo</span> <span class="s2">"http://www.google.com"</span>|awk <span class="s1">'{if(system("curl -o /dev/null -s -w %{http_code} "$1)~/^[2|3]/){print ":"$1 } else{ print ":http://www.baidu.com/"}}'</span>
200:http://www.baidu.com/
</code></pre>
</div>
<table>
<tbody>
<tr>
<td>思前想后,在百度大婶的帮助下,终于搞明白一个问题:system()的结果是直接返回给shell显示了,然后再由awk继续执行后面的程序,这种情况下,if()里留下的其实是system()的执行状态【即0或1】”0”~/^[2</td>
<td>3]/,当然就一直执行else了。</td>
</tr>
</tbody>
</table>
<p>糟糕的问题是awk的getline,无法直接把system()的执行结果导入awk的变量…除非我先system里>一个文件,然后getline<这个文件。MyGod!</p>
<table>
<tbody>
<tr>
<td>而如果采用while(“curl”</td>
<td>getline var)的执行方式,如何传递shell变量进去又成了问题……唉</td>
</tr>
</tbody>
</table>
squid3 的 gzip 支持
2009-12-30T00:00:00+08:00
squid
http://chenlinux.com/2009/12/30/squid-gzip
<p>标题很简单,内容很复杂,而且我也用不上。姑且存档吧:<br />
原标题《VIGOS eCAP GZIP Adapter for SQUID Proxy Cache》<br />
第一句话就让我很绝望,本插件只适用于squid3.1及以上版本——squid有比3.1更高的版本么——前文提到的有session控制的squid2.7我都还没看呢~哭</p>
<p>办法:<br />
<code class="highlighter-rouge">bash
wget http://www.measurement-factory.com/tmp/ecap/libecap-0.0.2.tar.gz
wget http://www.vigos.com/products/eCAP/vigos-ecap-gzip-adapter-1.1.0.tar.gz
wget http://www.squid-cache.org/Versions/v3/3.1/squid-3.1.0.9.tar.gz
tar xvfz squid-3.1.0.9.tar.gz
tar xvfz libecap-0.0.2.tar.gz
tar xvfz vigos-ecap-gzip-adapter-1.1.0.tar.gz
cd libecap-0.0.2/
./configure
make
make install
cd ../vigos-ecap-gzip-adapter-1.1.0/
./configure
make
make install
cd ../squid-3.1.0.9/
./configure --enable-ecap
make
make install
cat >> etc/squid.conf <<EOF
ecap_enable on
ecap_service gzip_service respmod_precache 0
ecap://www.vigos.com/ecap_gzip
loadable_modules /usr/local/lib/ecap_adapter_gzip.so
acl GZIP_HTTP_STATUS http_status 200
adaptation_access gzip_service allow GZIP_HTTP_STATUS
EOF
</code><br />
以上,3.1毕竟还是测试版,这个留待以后吧~~</p>
<p>另,据《squid中文权威指南》作者潘勇华的blog说,这个“基于squid3.1的eCAP接口的流量压缩适配器”,对被自己压缩过的内容,简单的把Etag删除。</p>
url_rewrite_program(squid游戏恶搞~)
2009-12-30T00:00:00+08:00
squid
http://chenlinux.com/2009/12/30/a-reverse-example-of-url_rewrite_program
<p>标题很引人吧。其实是我在查找squid的rewrite资料时,看到的一篇文章。原作者突发奇想,准备让公司的同事们上网时,看到的所有图片全都倒过来180°。嘿嘿,小样,还不拧断你们脖子~~</p>
<p>当然这么个大计划,单靠squid还是不行的,得后台配合一下web服务器才行。</p>
<p>原理是这样的:squid每接到一个请求,首先判断后缀名是不是图片类型,如果是,下载到web服务器目录,然后调用程序颠倒图片,拷贝去另一个发布路径下。最后把新的路径返回给squid,交给浏览器去看。</p>
<p>假设下载目录是/revimg,发布目录是/revimg/out,那么apache里配置如下vhost:<br />
<code class="highlighter-rouge">apache
ServerName revimg.soulogic.com
DocumentRoot /revimg/out
</code><br />
好了,其他准备完成。进入squid部分:</p>
<p>squid官方并没有转向的设定,不过他允许甚至推荐了一些转向外挂(好吧,说好听些,第三方插件)。只需要很简单的在squid.conf里启用url_rewrite_program(也叫redirect_program)就可以了:</p>
<p>url_rewrite_program /etc/squid/redirect.php(常见的是pl、py,当然也可以是squirm、squidGuard等软件,其实只要是squid属主可执行文件,parse都能通过)</p>
<p>根据需要,还有相关的children、access、host配置。<br />
下面是原作者提供的php代码。很赞,可惜还没看懂,慢慢品味:<br />
<code class="highlighter-rouge">php
#!/usr/bin/php
<?PHP
chdir("/revimg");
$sServer = "revimg.soulogic.com";
while ($sContent = fgets(STDIN) ) {
$sContent = trim($sContent);
if (empty($sContent)) {
continue;
}
$aArg = explode(" ", $sContent, 5);
$sURL = $aArg[0];
if ($aArg[3] != "GET") {
fwrite(STDOUT, $sURL."n");
continue;
}
$aURL = parse_url($sURL);
$aURL += array("scheme" => "", "host" => "", "path" => "");
if ($aURL["scheme"] != "http"
|| $aURL["host"] == $sServer
|| !preg_match("/^[0-9a-z\-]+(\.[0-9a-z\-]+)+(:[0-9]{2,5})?$/i", $aURL["host"])
|| !preg_match("/\.(jpg|jpeg|png|gif)$/i", $aURL["path"])
) {
fwrite(STDOUT, $sURL."n");
continue;
}
// 检测通过,处理图片
$sHash = md5($sURL);
$sDir = substr($sHash, 0, 2)."/".substr($sHash, 2, 2);
$sFile = $sDir."/".substr($sHash, 4);
$sFileOut = "out/".$sFile;
if (!file_exists($sDir)) {
mkdir($sDir, 0777, TRUE);
mkdir("out/".$sDir, 0777, TRUE);
chmod("out", 0777);
chmod("out/".substr($sHash, 0, 2), 0777);
chmod("out/".$sDir, 0777);
}
if (!file_exists($sFile)) {
$sCmd = "wget -qc ".escapeshellarg($sURL)." -O ".$sFile;
exec($sCmd);
}
if (!file_exists($sFileOut)) {
$sCmd = "convert ".$sFile." -flip -quality 80 ".$sFileOut;
exec($sCmd);
chmod($sFileOut, 0666);
}
$sURL = "<a href="http://&quot;/">http://"</a>.$sServer."/".$sFile;
fwrite(STDOUT, $sURL."n");
}
?>
</code><br />
原作者说:squid在重定向处理是采用的标准输入输出方式,所以测试的时候只需要cat test.txt|/etc/squid/redirector.php就可以了。<br />
这个东东还能进一步优化。因为图片在浏览器本地也有缓存的,如果之前同事们已经上过,怎么办?还得改写Expires。</p>
url_rewrite_program(首次访问跳转)
2009-12-30T00:00:00+08:00
squid
http://chenlinux.com/2009/12/30/a-location-example-of-url_rewrite_program
<p>上文提到,squid大多数的rewrite_program是用perl编写的。现在就转几个简洁明了的redirect.pl。虽说我至今没把perl基础教程看完。不过从中领会一下squid的rewrite流程,还是可以的:</p>
<p>例一:这是最简单的url转向,把http://www.baidu.com/转到https://www.google.com:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/perl -w</span>
<span class="vg">$|</span><span class="o">=</span><span class="mi">1</span><span class="p">;</span>
<span class="k">while</span> <span class="p">()</span> <span class="p">{</span>
<span class="nv">@X</span> <span class="o">=</span> <span class="nb">split</span><span class="p">;</span>
<span class="nv">$url</span> <span class="o">=</span> <span class="nv">$X</span><span class="p">[</span><span class="mi">0</span><span class="p">];</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$url</span> <span class="o">=~</span> <span class="nv">m</span><span class="c1">#^http://www.baidu.com#) {</span>
<span class="k">print</span> <span class="s">"302:http://www.google.com\n"</span><span class="p">;</span>
<span class="p">}</span> <span class="k">elsif</span> <span class="p">(</span><span class="nv">$url</span> <span class="o">=~</span> <span class="nv">m</span><span class="c1">#^http://gmail.com#) {</span>
<span class="k">print</span> <span class="s">"302:http://gmail.google.com\n"</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="k">print</span> <span class="s">"$url\n"</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>例二:这个的功效,是当第一次访问外网网站时,先跳转到公司主页一次,之后再访问就不再限制。</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="c1">#!/usr/bin/perl -w</span>
<span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">DB_File</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">vars</span> <span class="sx">qw (%cache </span><span class="nv">$uri</span> <span class="nv">$cachetime</span><span class="p">);</span>
<span class="vg">$|</span><span class="o">=</span><span class="mi">1</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$timeout</span><span class="o">=</span><span class="mi">3600</span><span class="p">;</span><span class="c1">#缓存超时时间</span>
<span class="k">while</span> <span class="p">(){</span>
<span class="k">my</span> <span class="p">(</span><span class="nv">$client</span><span class="p">,</span><span class="nv">$ident</span><span class="p">,</span><span class="nv">$method</span><span class="p">)</span><span class="o">=</span><span class="p">();</span>
<span class="p">(</span><span class="nv">$uri</span><span class="p">,</span><span class="nv">$client</span><span class="p">,</span><span class="nv">$ident</span><span class="p">,</span><span class="nv">$method</span><span class="p">)</span><span class="o">=</span><span class="nb">split</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">check</span><span class="p">(</span><span class="nv">$client</span><span class="p">)){</span>
<span class="k">next</span><span class="p">;</span>
<span class="p">}</span><span class="k">else</span><span class="p">{</span>
<span class="nv">save</span><span class="p">(</span><span class="nv">$client</span><span class="p">);</span>
<span class="nv">$uri</span><span class="o">=</span><span class="s">"301:http://www.xxxx.cn"</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span> <span class="k">continue</span> <span class="p">{</span>
<span class="k">print</span> <span class="nv">$uri</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">check</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$client</span><span class="o">=</span><span class="nb">shift</span><span class="p">;</span>
<span class="c1"># init cache time</span>
<span class="k">my</span> <span class="nv">$time</span><span class="o">=</span><span class="nb">time</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nv">$cachetime</span><span class="p">){</span>
<span class="nv">$cachetime</span><span class="o">=</span><span class="nv">$time</span><span class="o">+</span><span class="nv">$timeout</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$time</span> <span class="o">&</span><span class="ow">gt</span><span class="p">;</span> <span class="nv">$cachetime</span><span class="p">){</span>
<span class="nv">%cache</span><span class="o">=</span><span class="p">();</span>
<span class="nv">$cachetime</span><span class="o">=</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">return</span> <span class="mi">1</span> <span class="k">if</span> <span class="nv">$cache</span><span class="p">{</span><span class="nv">$client</span><span class="p">};</span>
<span class="c1"># reopen db</span>
<span class="k">my</span> <span class="nv">%ip</span><span class="o">=</span><span class="p">();</span>
<span class="nb">tie</span> <span class="nv">%ip</span><span class="p">,</span><span class="s">"DB_File"</span><span class="p">,</span><span class="s">"/tmp/ip.db"</span><span class="p">,</span><span class="nv">O_CREAT</span><span class="o">|</span><span class="nv">O_RDWR</span><span class="p">,</span><span class="mo">0666</span><span class="p">;</span>
<span class="nv">%cache</span><span class="o">=</span><span class="nv">%ip</span><span class="p">;</span> <span class="c1">#hard copy?</span>
<span class="nb">untie</span> <span class="nv">%ip</span><span class="p">;</span>
<span class="nv">$cache</span><span class="p">{</span><span class="nv">$client</span><span class="p">}</span> <span class="p">?</span> <span class="mi">1</span><span class="p">:</span><span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">sub </span><span class="nf">save</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$client</span><span class="o">=</span><span class="nb">shift</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">%ip</span><span class="o">=</span><span class="p">();</span>
<span class="nb">tie</span> <span class="nv">%ip</span><span class="p">,</span><span class="s">"DB_File"</span><span class="p">,</span><span class="s">"/tmp/ip.db"</span><span class="p">,</span><span class="nv">O_CREAT</span><span class="o">|</span><span class="nv">O_RDWR</span><span class="p">,</span><span class="mo">0666</span><span class="p">;</span>
<span class="nv">$ip</span><span class="p">{</span><span class="nv">$client</span><span class="p">}</span><span class="o">=</span><span class="mi">1</span><span class="p">;</span>
<span class="nb">untie</span> <span class="nv">%ip</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
</div>
squid_session(首次访问跳转)
2009-12-30T00:00:00+08:00
squid
http://chenlinux.com/2009/12/30/a-location-example-of-squid_session
<p>上文perl例二中提到的第一次上某网站时先跳转公司主页。在CU上看到另一种办法。即在squid2.7以上版本中,有个squid_session。操作如下:<br />
首先是安装:<br />
安装squid ( for 2.7 stable )<br />
修改源代码:vi src/errorpage.c 60行处将</p>
<p>“Generated %T by %h (%s)\n”</p>
<p>删除,让squid错误页面不产生服务器信息。<br />
<code class="highlighter-rouge">bash
mkdir -p /usr/local/squid
./configure
configure options: '--prefix=/usr/local/squid'
'--enable-storeio=diskd,ufs,aufs,null' '--enable-async-io=80'
'--enable-icmp' '--enable-removal-policies=heap,lru'
'--enable-useragent-log' '--enable-snmp' '--enable-referer-log'
'--enable-kill-parent-hack' '--enable-cache-digests'
'--enable-default-err-language=Simplify_Chinese'
'--enable-err-languages=Simplify_Chinese' '--enable-gnuregex'
'--enable-ipf-transparent' '--enable-pf-transparent'
'--enable-follow-x-forwarded-for' '--disable-wccp'
'--disable-delay-pools' '--disable-ident-lookups'
'--disable-arp-acl' '--with-large-files
make; make install; make clean
mkdir /usr/local/squid/helper
mkdir /usr/local/squid/che
cd hepler/external_acl/session; make
cp squid_session /usr/local/squid/helper
chmod 777 /usr/local/squid/che
chmod 777 /usr/local/squid/var
chmod 777 /usr/local/squid/var/logs
grep -v '^#' /etc/squid/squid.conf | sed -e '/^$/d' > /etc/squid/squid.conf.orig
mv /etc/squid.conf /etc/squid.conf.system
mv /etc/squid/squid.conf.orig /etc/squid/squid.conf
</code><br />
然后是squid.conf的修改:<br />
<code class="highlighter-rouge">squid
external_acl_type session ttl=300 negative_ttl=0 children=1
concurrency=200 %SRC /usr/local/squid/helper/squid_session -t 900
//客户端第一个网页转向
acl session external session
acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32
acl localnet src 10.0.0.0/8
# RFC1918 possible internal network
acl localnet src 172.16.0.0/12
# RFC1918 possible internal network
acl localnet src 192.168.0.0/16
# RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80
# http
acl Safe_ports port 21
# ftp
acl Safe_ports port 443
# https
acl Safe_ports port 70
# gopher
acl Safe_ports port 210
# wais
acl Safe_ports port 1025-65535
# unregistered ports
acl Safe_ports port 280
# http-mgmt
acl Safe_ports port 488
# gss-http
acl Safe_ports port 591
# filemaker
acl Safe_ports port 777
# multiling http
acl CONNECT method CONNECT
acl rangeget req_header Range .*
//定义多线程下载规则
http_access deny !session
//只有第一个执行才能打开
deny_info firstpage session
//deny_info指定的页面
http_access deny rangeget
//不允许多线程下载
http_access allow all
icp_access allow localnet
icp_access deny all
http_port 127.0.0.1:3128 transparent
http_port 192.168.101.1:3128 transparent
//绑定IP和端口,透明代理
http_port 192.168.188.1:3128 transparent
access_log /usr/local/squid/var/logs/access.log squid
//各种日志信息存放路径
cache_log /usr/local/squid/var/logs/cache.log
cache_store_log /usr/local/squid/var/logs/store.log
pid_filename /usr/local/squid/var/squid.pid
coredump_dir /usr/local/squid/var/coredump
cache_mem 100MB
//使用内存 总内存的一半
maximum_object_size_in_memory 40KB
//内存中对像大小
cache_swap_low 90
cache_swap_high 95
cache_dir aufs /usr/local/squid/che 500 16
256 //缓存目录
hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin ?
cache deny QUERY
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|?) 0 0% 0
refresh_pattern . 0 20% 4320
acl shoutcast rep_header X-HTTP09-First-Line ^ICY.[0-9]
upgrade_http0.9 deny shoutcast
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
header_access Via deny all
header_access X-Cache deny all
//隐藏服务器信息
header_access X-Cache-Lookup deny all
header_access X-Forward-For deny all
via off
// 隐藏服务器信息
check_hostnames on
//检查主机名称
allow_underscore
//允许出现下划线
logfile_rotate 4
//rotate后保存日志数量
cache_mgr rainren_openbsd@yahoo.cn
visible_hostname rain
httpd_suppress_version_string on
// 隐藏服务器信息
</code><br />
最后创建ERR页,即squid.conf中的/usr/local/squid/share/errors/Simplify_Chinese/firstpage,如下:<br />
```html<br />
<meta http-equiv=”refresh” content=”10;url=”http://www.google.com/”></p>
<meta http-equiv="Content-Type" content="text/html;charset=gb2312" />
<p>rainren<br />
legend {<br />
font-size:18px;<br />
font-weight:bold;<br />
color:black;<br />
}<br />
p {<br />
font-size:16px;<br />
color:#FF0000;<br />
}<br />
```<br />
热诚欢迎您<br />
怎么样?<br />
好了,以上都是转载,还没试验过,不过有一个我知道的,就是不用修改ERR页,在deny_info里,可以直接写http://www.google.com。</p>
nginx502错误
2009-12-30T00:00:00+08:00
nginx
http://chenlinux.com/2009/12/30/502-error-in-nginx
<p>NGINX 502 Bad Gateway错误是FastCGI有问题,造成NGINX 502错误的可能性比较多。将网上找到的一些和502 Bad Gateway错误有关的问题和排查方法列一下,先从FastCGI配置入手:</p>
<ol>
<li>查看FastCGI进程是否已经启动</li>
</ol>
<p>NGINX 502错误的含义是sock、端口没被监听造成的。我们先检查fastcgi是否在运行</p>
<ol>
<li>检查系统Fastcgi进程运行情况</li>
</ol>
<p>除了第一种情况,fastcgi进程数不够用、php执行时间长、或者是php-cgi进程死掉也可能造成nginx的502错误<br />
运行以下命令判断是否接近FastCGI进程,如果fastcgi进程数接近配置文件中设置的数值,表明worker进程数设置太少<br />
netstat -anpo | grep “php-cgi” | wc -l</p>
<ol>
<li>FastCGI执行时间过长</li>
</ol>
<p>根据实际情况调高以下参数值</p>
<p>fastcgi_connect_timeout 300; <br />
fastcgi_send_timeout 300; <br />
fastcgi_read_timeout 300;</p>
<ol>
<li>头部太大</li>
</ol>
<p>nginx和apache一样,有前端缓冲限制,可以调整缓冲参数</p>
<p>fastcgi_buffer_size 32k;<br />
fastcgi_buffers 8 32k;</p>
<p>如果你使用的是nginx的负载均衡Proxying,调整</p>
<p>proxy_buffer_size 16k;<br />
proxy_buffers 4 16k;</p>
<ol>
<li>https转发配置错误</li>
</ol>
<p>正确的配置方法<br />
<code class="highlighter-rouge">nginx
server_name www.xok.la;
location /myproj/repos {
set $fixed_destination $http_destination;
if ( $http_destination ~* ^https(.*)$ ) {
set $fixed_destination http$1;
}
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Destination $fixed_destination;
proxy_pass http://subversion_hosts;
}
</code></p>
看上去很美——国内CDN现状与美国对比(转)
2009-12-27T00:00:00+08:00
CDN
http://chenlinux.com/2009/12/27/zz-diff-cdn-between-china-and-usa
<p>遵嘱注明:本文转载自:<a href="http://www.php-oa.com">扶凯</a><br />
原链接:<a href="http://blog.csdn.net/fayu0903/archive/2009/03/25/4022979.aspx">http://blog.csdn.net/fayu0903/archive/2009/03/25/4022979.aspx</a></p>
<p>作者:阀域</p>
<ul>
<li>CDN的理想与现实</li>
</ul>
<p>多年以前,当《Kingdom of Heaven》这部史诗电影发行的时候,中国的影迷使用电驴和BT来寻找种子,而那个时候,高清也才刚刚进入电影领域,我的同事不惜用自家的电脑花费一个星期的时间去下载高清的版本。而现在,中国的影迷在使用迅雷去下载《越狱》,而每一集越狱播出以后,在20小时之内,迅雷上面就可以下载到有中文字幕的完整版本,而影迷只要半个小时就可以下载完成,他们使用的是以“CDN”为基础的所谓P2SP服务。我们在这里不需要讨论盗版的问题,我们在这里想谈论的只是互联网和CDN的改变……</p>
<ul>
<li>我们要谈论CDN么?</li>
</ul>
<p>互联网的改变让CDN变得不那么神秘与高深</p>
<p>CDN是个古老的东西,在互联网发展之初就已经出现了。一群MIT的精英份子发现如果要让任何地方的人都可以很快的打开自己的网站的话,就需要象在世界各地盖教堂一样,把自己的网页发布到离信众最近的地方去。所以,他们用一种简单的缓存镜像的办法实现了这种发布。最早的入主这个教堂网络的是Yahoo!那是在1998年。就像天做良缘,Yahoo!使用了当时世界上最大的CDN网络,当然现在他还是最大的。啊,忘了解释:CDN是内容投递网络(Content Delivery Network)的简称。我们可以在Wiki上面轻易找到这个单词的解释,但并不是所有人都能轻易理解CDN和它的意义,因为它是一个架接在互联网与传统电信运营商之间的看不见的桥梁。</p>
<p>让我们继续回顾互联网与CDN在那耀眼的一瞬之后的日子,互联网经历了泡沫的破裂,新模式的创新,但CDN却好像渐渐的被人遗忘了。我们熟悉的portal,垂直portal,鼠标加水泥,B2B,C2C,B2C,P2P,Web2.0,搜索,竞价排名,Page Ranking, RSS,Wiki,Meshup,podcasting,网络游戏,Social Network……那么多我们耳熟能详的名词,或者其实并不明白其中的真正意思,但在炫耀自己的互联网阅历的时候随口说出的几个里面,唯独没有CDN……直到短视频的出现。</p>
<p>当YouTube出现在世人面前的时候,人们为互联网的又一次革命而叫好,而与此同时,人们看到了在YouTube后面的一个强大的CDN支持,是这个CDN网络把YouTube的无数视频展现在人们面前,在这个时候,人们发现CDN是不可或缺的,CDN在经历了那么长时间的默默无闻以后,突然一夜间闻达于诸侯,就象君士坦丁大帝把天主教定为国教一样,大家突然认识到了一个不为人熟知的领域。但我们看到的是什么呢?君士坦丁介绍给罗马的天主教是耶稣创立时的教义么?我们所看到的CDN是MIT创立时的CDN么?</p>
<p>人们开始搜索CDN,研究CDN,发现CDN是那么的简单,可以用一页PPT就把原理讲的清清楚楚,而网络硬件的厂商也会这样和互联网的客户说,我们可以提供完整的CDN解决方案,你不需要做什么,买我们的硬件,它已经能够解决你所有的CDN问题。</p>
<p>从此,CDN变成了一个流行词汇,尤其是在高盛领投LimeLight(全球第二大CDN公司)1.2亿美金之后,突然之间,世界各地都出现了大大小小的CDN公司,无数的投资蜂拥而至,就像那时的罗马,人人都开始信仰天主教,也许是真的信仰,也许是为了圣餐,也许是为了研究,总之,“我们都爱CDN”。</p>
<p>也许有人问:我们谈论的是同一个CDN么,或者我们谈论的不是CDN?</p>
<p>当我们在说CDN的时候,所有的公司都是谈到两个偶像,就像谈耶稣和圣母一样,一个是Akamai(世界第一大CDN公司),一个就是LimeLight。所以,阵营就分开来了,要么介绍自己是师从Akamai,要不就说自己是LimeLight的真传弟子,尽管大部分这么说的人几乎没有见过Akamai和LimeLight的网络和服务,但并不影响大家对自己的夸耀和标榜。而CDN是什么却越来越没有人关注,哪怕是LimeLight和Akamai的区别也被人忽略了。我们每一个谈论CDN的人谈论的是同一个概念么?</p>
<p>CDN是有专利的,这一点与天主教的圣经不同,CDN的解释是可以通过查询这些文件发现的。CDN在利用DNS的转授权来引导最终访问者找到最理想的缓存或者镜像站点,它是基于域名的一种服务。在不同的实现方式下,最终的定位到哪个缓存和镜像站点的策略有很大的不同。Akamai使用的是传统的基于地理位置的定位策略,在世界各地的ISP里面,都会有自己的节点,而通过智能DNS的判断,可以为用户找到离自己地理位置“最近”的节点。而LimeLight则用的是完全不同的策略,LimeLight有自己的骨干网,给访问者的节点并不是地理位置“最近”的节点,而是路由层面“最近”的节点,这一点有点像我们访问网站不通过域名而直接通过IP访问一样,它会寻找对于访问者的ISP最近的路由是哪里,用那里的节点服务于这个访问。LimeLight的策略已经在很大程度上改变了CDN的工作方式,所以,当LimeLight准备上市的时候,谁都会认为他们已经绕过了Akamai的专利壁垒,但不幸的是,之后他们还是遇到了诉讼的麻烦,或许因为DNS转授权是无法改变的……</p>
<p>在这样的壁垒下面,任何做CDN的公司都好像要面对宗教法庭一样,要么被烧死,要么就皈依。所以,有好事者想我们如果使用其他的方式做CDN该有多好?我们既然可以把驴弄进贵州,为什么不能把P2P融进CDN?我记得一位Akamai的高管对我说过,那不是CDN,CDN是透明的……,所以,让我们忘了P2PCDN吧……。也许还没有人搞清楚P2P和CDN的关系,那么Cloud呢?也许是个好主意。但实际上客户已经在自己发展他们认为的CDN了,其中也包括所谓的P2P CDN。</p>
<p>那么CDN真的象我们想象的那么美好么?就像天国王朝里面的圣地?</p>
<p>当年的《天国王朝》,无数的十字军涌向耶路撒冷,那里是天国,是一个可以让灵魂升华与遍地黄金的地方,但生活在圣城的人们却发现事实与理想相去甚远……,现在人们涌向CDN,是因为它是一个看起来很美的行业,但实际上如何呢?</p>
<p>自从CDN成为一个业内的大众词汇,CDN服务就像卖白菜一样了,几乎没有人关心你的CDN和别人的CDN有什么差别,只是问多少钱1M,多少钱1G,好用么?回答的也是那么的合乎情理:你可以试试,不好不要钱。</p>
<p>另一方面,CDN客户的流失率从来没有下过10%,高的公司可以达到20%-30%,测试客户更是今天测,明天走。正是因为这样,客户基本上找不到满足要求的CDN公司,从而让很多人开始质疑CDN本身有问题,甚至突然觉得CDN应该是一个夕阳产业。</p>
<p>好在中国但凡叫得出名字的CDN公司,这几年的收入都是翻翻的,虽然利润少的可怜,而且那些利润也不是从CDN业务中获得的……</p>
<p>但我们同时发现在美国的Akamai却有着不同的表现,2007年6亿美金的销售额,1亿美金的纯利润,毛利更是超过40%,2008年至少有30%的成长。难道美国是CDN的天国,而中国就是被异教践踏的土地?还是我们并没有看到CDN真正的一面?就像柏拉图所描绘的山洞,我们看到只是火光照耀的影子?</p>
<p>事实上,互联网出现以来,只有CDN是没有海外公司进入中国市场的互联网业务,而正是这样的安排或者壁垒,让中国的CDN与海外的CDN有着巨大的差别。</p>
<ul>
<li>美国的CDN与中国CDN的对比</li>
</ul>
<p>在美国,CDN领域里面会有这样一些分类:静态内容的加速,动态内容的加速,大文件下载加速;对不同的客户类型,还会有不同的系统与之对应,比如SSL加速,Long Tail加速,Streaming加速;而对不同行业客户也会有不同的加速系统,比如媒体类客户加速,电子商务类客户加速,软件与IT行业客户的加速等等。甚至于对不同的客户规模也会有不同的系统与之对应:大流量客户的加速,中小客户的加速,甚至个人客户的加速;CDN的系统是一个庞杂而专业性非常强的领域。在这些领域与系统中,所有的功能甚至网络都是不一样的,配置的系统也是不一样的。</p>
<p>但在中国,这些系统的差别大部分是在售前的嘴里和不知所云的白皮书里面。而网络都是一个,功能都是一个,实现方式也是一个,所以就会出现如果一家CDN公司做不了一个功能,几乎所有的CDN公司都做不到,因为大家都是用最“通用的方式构建自己的通用CDN”,从而使中国CDN成为一个从电信转卖带宽的代理商。</p>
<p>我们发现了几个有趣的小例子:</p>
<p>中国的网站很注意防盗链(虽然并不注意防盗版),但是居然没有哪家CDN公司可以提供一个让客户满意的防盗链的系统,虽然各家都在说自己提供防盗链。就象我的一个做远程教育的朋友和我说的一样,“测试了能够叫得出名字的所有的CDN公司,但却没有一家CDN公司可以做好防盗链。”但在Akamai,这是一个很久就标准化的服务了。</p>
<p>中国是一个游戏和软件下载的大国,互联网上的主要流量是下载,而直到最近,国内的CDN公司才开始可以提供基于HTTP下载成功数的统计功能,而且还不是全自动的,是需要客户配合设置才可以使用。同样,在Akamai,这也是基本服务项目。</p>
<p>再有一个例子与视频有关了,短视频网站使用的Flash视频,在用户端是和文件下载没有区别的,用户会尽可能快的去下载完视频文件,而通常播放一个视频只需要300K码流就可以流畅播放,这样会有两种情况导致资源的浪费,第一、用户看视频并没有看完,但下载已经下载完了;第二、用户如果是宽带接入的话,虽然只需要300K带宽,可实际上却使用了1M。对于最终用户来说,这两点几乎不会有什么实际影响,但对视频网站来说,这意味着浪费了宝贵的带宽资源,在同等条件下支付了更多的成本。这与上面谈到的HTTP下载成功数计费是一个道理。而在美国,CDN公司是可以控制每个HTTP链接的速度,比如在开始播放的前30秒,1分钟不进行限速,而超过这个时间,就可以把速度控制在需要的范围内,以节省带宽资源。</p>
<p>至于SLA(服务等级协议),就更加的有趣,通常在中国的SLA是不会作为依据的,而评估好坏的标准是“你自己上网看看就知道了么”,这是中国现在一家发展很快的CDN公司的老总的看法。奇怪的是,使用CDN之后,没有哪家客户有能力在去进行所有地区的测试,看一下自己的网站是否比原来快。在这样的情况下,有些做论坛的客户就会发动自己在各地的版主进行测试,收集意见,然后再告诉CDN公司,你们哪里哪里不好,能否调一下?我们的CDN服务商然后说,“噢,你先给我们解释你是怎么测试的?给我一个你们测试的IP,我看看是不是不是中国的IP啊,千万不要给我你自己机器的IP,要给我LocalDNS的IP……,不知道LocalDNS?怎么连这个都不知道呢?……那是……”(其实我个人觉得还应该问问版主是不是中国人,也许这样更容易发现问题),我只能说,这是多好的客户啊……,在美国的CDN服务商会感动的掉泪的。</p>
<p>在美国,用户会每天得到一份SLA报告,会标出在什么时间段SLA没有达到标准,如果用户需要,还会给用户具体哪个区域没有达到SLA标准,而所有的这些,都只需要用户登录到BOSS的portal上面。</p>
<p>类似的例子还有很多,比如流的点播加速,长尾市场,小图片的加速等等。这些服务功能的差别也许还不是最主要的,而成本的控制与自动化的运营却是CDN公司能否盈利的关键,Akamai部署一个客户只需要10分钟,而国内部署需要客户在填写复杂的表格后,耐心的等待1天时间;Akamai管理上万台服务器只需要4个人,任何时间10%的服务器宕机都无需处理,因为系统会自动保证服务的可靠运行并自主恢复,而国内最小的CDN公司运维人员也有几十个,并且疲于应付各种“突发”事件。</p>
<p>现在,也许我们应该提一个建议,开放中国的CDN市场,让大部分中国的网站都可以看到真正的CDN服务是什么样子。</p>
<p>到这里,其实我们忽略了一个重要的问题,需求!中国的客户使用CDN很多时候是希望以此解决南北互通的问题,而美国客户没有这个问题,他们使用CDN首要考虑的是off load和降低成本。在有人发现有其他更加经济实惠的方案之后,CDN在米国的日子好像也不好过了。</p>
<ul>
<li>CDN被逐出了圣地?</li>
</ul>
<p>BT的出现对媒体行业来说是打开了一个盒子(也许是潘多拉,也许是宝盒),盗版的发行比以往任何时候都要快,成本也更低;而同时,视频直播也达到了前所未有的低成本,我们也许还记得在P2P客户端上看欧洲杯,看NBA,看奥运,如果是换成CDN,任何公司都会无法负担。</p>
<p>在看看客户情况,伟大的Google是一个什么事情都要insource的公司,它从来不使用CDN,但它的服务遍及全球,Amazon的EC2,SaleForce的CRM系统,SecondLife的虚拟世界,他们都没有在CDN上,但他们同样出色,而且看起来更有效率,成本更低。</p>
<p>现在,已经没有人会考虑使用CDN做大规模的直播服务了,充其量是作为一个备份手段;而自从YouTube离开LimeLight以后,CDN的光环也开始慢慢退去,VC和投机者开始又一股脑的质疑这个奇怪的生意,CDN有价值么?为什么盈利这么困难呢?就像勇猛的萨拉丁赶走了十字军一样,难道CDN会被P2P和不断升级的光纤所取代?还是象经过改革的宗教一样,即使历经文艺复兴与科技的反复冲击,而今依然影响着无数的人们。</p>
<ul>
<li>CDN的宗教革命?</li>
</ul>
<p>我们之所以把CDN比作宗教,是因为CDN到现在也有很多“流派”,LimeLight的大节点,大带宽的做法被许多IDC与运营商背景的公司所推崇,而传统的Akamai模式则是独立于运营商之外的CDN公司所首选的道路;而各种新奇理念的出现更是让CDN行业象是一个万花筒,从而也使其拥有更多的互联网气息。Amazon的CloudFront,EC2;Level 3的ITM;Prime的CDN Aggregation,CDTM;SimpleCDN的S3+等等,如此众多的演变,任何一种都是对传统CDN,甚至是对LimeLight模式的革命。感谢LimeLight与Akamai的成功,让很多优秀的工程师与天才的梦想家投入到CDN这个被Akamai一家统治多年的领域,并不断给我们与CDN客户以惊喜。但当我们把目光从美国看回中国的时候(上述所有的公司都是美国企业),我们要面对的是什么样的现实?</p>
<p>中国的CDN是CDN的佛教还是道教?还是象柏阳说得被中国这个大染缸去其精华留其糟粕的垃圾?</p>
<p>说道中国的CDN,我们可能要问:什么时候CDN开始没人关注SLA?什么时候CDN开始不提供标准的95/5计费?什么时候,全网配置的服务被悄悄替换成了“部分”节点配置?什么时候,CDN变成了一个黑匣子,客户无从了解自己的服务与问题,也无法控制自己的内容的发布与刷新(美国的CDN客户是可以直接自己设置3段TTL时间的)。这些我们无从而知,因为可能从我们开始认识中国CDN的时候就已经是这个样子了。</p>
<p>所以这些后来的中国的CDN厂商第一件要做的事情就是鼓吹自己的节点数量,而不管是否这些节点都为所有客户提供服务(客户甚至不知道有几个节点在为自己服务。PS:一般情况下,中国CDN公司为每个客户配置的节点不会超过20个)</p>
<p>所以,当海外的CDN公司在网站上介绍其服务的时候,中国客户通常很难找到他们全球有多少节点的信息,而中国CDN公司则乐此不疲的修改自己的节点地图,也不管地图画得就像一张地雷分布图。而销售人员更是信口开河的讲自己的公司有几百个节点,可笑的是,几乎所有的CDN公司都有“几百”个节点,全中国的IDC恐怕都不够这些CDN公司瓜分的了。</p>
<p>中国CDN已经把CDN本土化了,经常挂在嘴边的是外国公司不了解中国的网络,也许是吧,但到现在为止,没有一家本土CDN公司可以解决教育网的服务质量问题。而大部分的客户到CDN节点机房看到的情况是:“哦,怎么你们在使用和我们自己一样的系统?!”这是为什么呢?因为中国的CDN公司认为CDN是运营业务,就像中国移动,中国电信一样,最主要的是运营;</p>
<p>而象Akamai这样的美国CDN公司首先认为自己是技术公司,然后才是运营公司,Akamai的系统从底层到应用是自己开发的,所有的服务是自己开发的,所有的控制与监管是自己开发的,甚至在早期,连硬件都要自己开发。我记得一位VC界的著名人物这样评价在中国很出名的CDN企业:“他们没有技术”。这就是中国CDN与美国CDN公司的最大差距。那我们要问,难道十几年中国CDN的发展就白费了么?当然不是的,中国的CDN在花费了大量的时间在处理用户的客户化需求,可惜这些需求主要表现在计费领域,有的CDN公司居然有上百种计费方式。</p>
<p>而对于CDN的黑盒子问题(即用户看不到及时全面的数据),就像佛教的禅宗一样,一句“不可说”,客户也没有任何办法。而对于道教的无为则通常会用在对付客户的投诉上面。当然,这是不能责怪CDN的运营人员的,即使在CDN如火如荼的年代,能够说清楚CDN的具体情况与细节的人也还是少之又少。记得上面提到的发展很快的那家CDN公司的老总说到视频:“我们觉得视频是一个很简单的服务,根本没有难度”,但当客户测试他们视频服务,却选择了其他运营商之后,已经很少听到这种气壮山河的言论了。</p>
<p>中国的CDN虽然经历了很长的时间,但却没有真正的积累下来,而本土化,或者异化的CDN使本来就难于理解的CDN服务被断章取义的成为一个Cache和带宽的替代词汇。</p>
<p>而事实上,真正的CDN服务或者CDN的本来面目我们也许就从没有见到,就像中国的网站编程,在IE下看的好好的,但是换了浏览器就全乱了,这才发现原来没有按照W3C标准编写,而问题是很多的编程人员根本就没看过这些标准,所以也就不知道原来IE并不是严格遵照W3C解析与渲染的。</p>
<div class="highlighter-rouge"><pre class="highlight"><code>To be, or not to be: that is the question!
</code></pre>
</div>
<p>当莎士比亚的这句名言作为出现在这里的时候,我们应该考虑的是CDN的未来,还是中国CDN的未来?</p>
<p>让我们在更高的层面来看待如今的年代,越来越多的闪光点出现在这个领域,就像在Google出现之前,没有人关注搜索一样,CDN有可能成为第二个孕育奇迹的行业。CDN伴随着互联网的成长起起落落,现在的服务已经不仅仅局限在内容的分发,越来越多的CDN公司开始提供以复杂分布式存储为核心的存储网络服务,Amazon的S3是一个典型的代表,而Prime的FileGrid则是另一个值得关注的方向。Amazon在这个领域已经耕耘多年,虽然不是一个CDN公司,但实际上的服务内容已经涵盖了CDN服务,而其基于运计算的EC2服务更是现在可以使用的唯一的“云”了。有人说云计算就是CDN的未来,这也许还很远,Bill Gate的话也许更有道理:“云存储离我们更近一些”。无论是云计算还是云存储,对于CDN公司来说,都要比其他任何行业的公司都要靠近云端,而CDN的路线图也一定会是这样的发展,从内容网络到存储网络再到计算网络,而未来的CDN也会象使用电灯一样容易。</p>
<p>对于CDN的未来憧憬,另外的方向就是CDN云,尽管只是一个概念,但确实是一个很宏伟的想法,虽然现在可以看到的服务只有Prime的CDTM,或许这是CDN云的初级阶段──一个利用众多的CDN网络构建一个更高效成本控制更好与更智能的网络,但谁能够忽视这令人兴奋的进步与想象力?</p>
<p>在中国,各家CDN公司都开始大力开发自己的产品,也从没有象现在这样重视研发。而Akamai,CDNetwork已经开始进入中国;Prime也开始在中国展开一些试探。中国的客户已经开始体验到不同的服务,有的CDN公司提出的SLA至少看起来已经是一份有价值或者是可以度量的标准文件了。虽然路还是有些漫长,但相信中国的客户将会很快体验到世界水准的CDN服务,以及天才工程师们所创造的更多令人兴奋的网络产品。</p>
<p>中国乃至世界CDN领域的大变革正发生在我们身边,也许明天你就会看到一个不同的CDN来到你的眼前。</p>
查看一些常见应用的编译选项
2009-12-13T00:00:00+08:00
linux
http://chenlinux.com/2009/12/13/show-some-common-apps-compiler-options
<ul>
<li>
<p>nginx:<br />
<code class="highlighter-rouge">bash
[rao@localhost ~]$ /home/nginx/sbin/nginx -V|grep conf
nginx version: 0.7.54
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-44)
configure arguments: --prefix=/home/nginx --with-pcre
--with-http_stub_status_module --without-http_memcached_module
--without-http_fastcgi_module
apache:
[rao@localhost ~]$ cat /home/apache2/build/config.nice
#! /bin/sh
#
# Created by configure
CFLAGS="-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64"; export
CFLAGS
"./configure"
"--prefix=/home/apache2"
"--enable-static-support"
"--enable-rewrite"
"--with-mpm=worker"
"--enable-logio"
"--enable-so"
"--enable-mime-magic"
"--disable-cgid"
"--disable-cgi"
"--disable-userdir"
"--disable-dir"
"--disable-include"
"--disable-filter"
"--disable-env"
"--disable-setenvif"
"--disable-status"
"--disable-autoindex"
"--disable-asis"
"--disable-alias"
"--disable-actions"
"--disable-authn-file"
"--disable-authn-default"
"--disable-authz-default"
"--disable-authz-groupfile"
"--disable-authz-user"
"--disable-auth-basic"
"CFLAGS=-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64"
"$@"
</code></p>
</li>
<li>
<p>php:</p>
</li>
</ul>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">[</span>rao@localhost ~]<span class="nv">$ </span>php -i|grep configure
Configure Command <span class="o">=</span>> <span class="s1">'./configure'</span>
<span class="s1">'--build=x86_64-redhat-linux-gnu'</span> <span class="s1">'--host=x86_64-redhat-linux-gnu'</span>
<span class="s1">'--target=x86_64-redhat-linux-gnu'</span> <span class="s1">'--program-prefix='</span>
<span class="s1">'--prefix=/usr'</span> <span class="s1">'--exec-prefix=/usr'</span> <span class="s1">'--bindir=/usr/bin'</span>
<span class="s1">'--sbindir=/usr/sbin'</span> <span class="s1">'--sysconfdir=/etc'</span> <span class="s1">'--datadir=/usr/share'</span>
<span class="s1">'--includedir=/usr/include'</span> <span class="s1">'--libdir=/usr/lib64'</span>
<span class="s1">'--libexecdir=/usr/libexec'</span> <span class="s1">'--localstatedir=/var'</span>
<span class="s1">'--sharedstatedir=/usr/com'</span> <span class="s1">'--mandir=/usr/share/man'</span>
<span class="s1">'--infodir=/usr/share/info'</span> <span class="s1">'--cache-file=../config.cache'</span>
<span class="s1">'--with-libdir=lib64'</span> <span class="s1">'--with-config-file-path=/etc'</span>
<span class="s1">'--with-config-file-scan-dir=/etc/php.d'</span> <span class="s1">'--disable-debug'</span>
<span class="s1">'--with-pic'</span> <span class="s1">'--disable-rpath'</span> <span class="s1">'--without-pear'</span> <span class="s1">'--with-bz2'</span>
<span class="s1">'--with-curl'</span> <span class="s1">'--with-exec-dir=/usr/bin'</span> <span class="s1">'--with-freetype-dir=/usr'</span>
<span class="s1">'--with-png-dir=/usr'</span> <span class="s1">'--enable-gd-native-ttf'</span> <span class="s1">'--without-gdbm'</span>
<span class="s1">'--with-gettext'</span> <span class="s1">'--with-gmp'</span> <span class="s1">'--with-iconv'</span> <span class="s1">'--with-jpeg-dir=/usr'</span>
<span class="s1">'--with-openssl'</span> <span class="s1">'--with-png'</span> <span class="s1">'--with-pspell'</span>
<span class="s1">'--with-expat-dir=/usr'</span> <span class="s1">'--with-pcre-regex=/usr'</span> <span class="s1">'--with-zlib'</span>
<span class="s1">'--with-layout=GNU'</span> <span class="s1">'--enable-exif'</span> <span class="s1">'--enable-ftp'</span>
<span class="s1">'--enable-magic-quotes'</span> <span class="s1">'--enable-sockets'</span> <span class="s1">'--enable-sysvsem'</span>
<span class="s1">'--enable-sysvshm'</span> <span class="s1">'--enable-sysvmsg'</span> <span class="s1">'--enable-track-vars'</span>
<span class="s1">'--enable-trans-sid'</span> <span class="s1">'--enable-yp'</span> <span class="s1">'--enable-wddx'</span>
<span class="s1">'--with-kerberos'</span> <span class="s1">'--enable-ucd-snmp-hack'</span>
<span class="s1">'--with-unixODBC=shared,/usr'</span> <span class="s1">'--enable-memory-limit'</span>
<span class="s1">'--enable-shmop'</span> <span class="s1">'--enable-calendar'</span> <span class="s1">'--enable-dbx'</span> <span class="s1">'--enable-dio'</span>
<span class="s1">'--with-mime-magic=/usr/share/file/magic.mime'</span> <span class="s1">'--without-sqlite'</span>
<span class="s1">'--with-libxml-dir=/usr'</span> <span class="s1">'--with-xml'</span> <span class="s1">'--with-system-tzdata'</span>
<span class="s1">'--enable-force-cgi-redirect'</span> <span class="s1">'--enable-pcntl'</span> <span class="s1">'--with-imap=shared'</span>
<span class="s1">'--with-imap-ssl'</span> <span class="s1">'--enable-mbstring=shared'</span>
<span class="s1">'--enable-mbstr-enc-trans'</span> <span class="s1">'--enable-mbregex'</span>
<span class="s1">'--with-ncurses=shared'</span> <span class="s1">'--with-gd=shared'</span> <span class="s1">'--enable-bcmath=shared'</span>
<span class="s1">'--enable-dba=shared'</span> <span class="s1">'--with-db4=/usr'</span> <span class="s1">'--with-xmlrpc=shared'</span>
<span class="s1">'--with-ldap=shared'</span> <span class="s1">'--with-ldap-sasl'</span> <span class="s1">'--with-mysql=shared,/usr'</span>
<span class="s1">'--with-mysqli=shared,/usr/bin/mysql_config'</span> <span class="s1">'--enable-dom=shared'</span>
<span class="s1">'--with-dom-xslt=/usr'</span> <span class="s1">'--with-dom-exslt=/usr'</span>
<span class="s1">'--with-pgsql=shared'</span> <span class="s1">'--with-snmp=shared,/usr'</span>
<span class="s1">'--enable-soap=shared'</span> <span class="s1">'--with-xsl=shared,/usr'</span>
<span class="s1">'--enable-xmlreader=shared'</span> <span class="s1">'--enable-xmlwriter=shared'</span>
<span class="s1">'--enable-fastcgi'</span> <span class="s1">'--enable-pdo=shared'</span>
<span class="s1">'--with-pdo-odbc=shared,unixODBC,/usr'</span>
<span class="s1">'--with-pdo-mysql=shared,/usr'</span> <span class="s1">'--with-pdo-pgsql=shared,/usr'</span>
<span class="s1">'--with-pdo-sqlite=shared,/usr'</span> <span class="s1">'--enable-dbase=shared'</span>
</code></pre>
</div>
<ul>
<li>mysql:<br />
据网上都说是cat /usr/bin/mysqlbug|grep conf,但我这的结果是压根没用……</li>
</ul>
引号的魔力
2009-12-04T00:00:00+08:00
bash
awk
http://chenlinux.com/2009/12/04/magic-of-quotes
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">[</span>root@neteasesquid1 ~]# <span class="nv">i</span><span class="o">=</span>1.2.3.4;awk -v <span class="nv">OFS</span><span class="o">=</span><span class="s2">"tt"</span> <span class="s1">'BEGIN{print '</span><span class="s2">"</span><span class="nv">$i</span><span class="s2">"</span><span class="s1">',"'</span><span class="s2">"</span><span class="nv">$i</span><span class="s2">"</span><span class="s1">'","'</span><span class="nv">$i</span><span class="s1">'",""'</span><span class="nv">$i</span><span class="s1">'""}'</span>
1.20.30.4
1.2.3.4
1.2.3.4
1.20.30.4
</code></pre>
</div>
<p>以前用awk调用shell变量时,一般都是文字字符串,看不出什么问题来;今天突然用上ip,发现输出结果显示不正常。于是做了如上实验。<br />
但是原因嘛,还是不知道……<br />
继续做下一个实验:<br />
<code class="highlighter-rouge">bash
[rcl@ubuntu:/win/learning/myshell]$ i=1.2.3;echo "1.2.3.4"|awk '/'"$i"'/{print}'
1.2.3.4
[rcl@ubuntu:/win/learning/myshell]$ i=1.2.3;echo "1.2.3.4"|awk '/"'$i'"/{print}'
[rcl@ubuntu:/win/learning/myshell]$ i=1.2.3;echo "1.2.3.4"|awk '/'"$i"'/{print "'$i'"}'
1.2.3
[rcl@ubuntu:/win/learning/myshell]$ i=1.2.3;echo "1.2.3.4"|awk '/'"$i"'/{print '"$i"'}'
1.20.3
[rcl@ubuntu:/win/learning/myshell]$ i=1.2.3;echo "1.20.3.4"|awk '/'"$i"'/{print}'
[rcl@ubuntu:/win/learning/myshell]$ i=1.2.3;echo "1.20.3.4"|awk '/"'$i'"/{print}'
</code><br />
所以最终结果,在regex里,awk引用shell变量是’“$i”‘,而在’{}’里则要写成”‘$i’“。乱呀……</p>
sed子命令集
2009-11-23T00:00:00+08:00
bash
sed
http://chenlinux.com/2009/11/23/sub-commands-in-sed
<p>1.:<br />
用法:<br />
:lable<br />
在脚本中标记一行,用于实现由b或t的控制转移。Label最多可以包含7个字符</p>
<p>2.=<br />
用法:<br />
=[address]=<br />
将所寻址的行编写到标准输出</p>
<p>3.a<br />
用法:<br />
[address]a<br />
text<br />
在与address匹配的每行后面追加text。如果text多于一行,必须用反斜杠将这些行前面的换行符“隐藏”起来。Text将被没有用这种方法隐藏的第一个换行符结束。Text在模式空间中是不可用的并且后续的命令不能应用于它。当编辑命令的列表用完时,这个命令的结果将被输送到标准输出,而不管在模式空间中的当前行发生了什么。</p>
<p>4.b<br />
用法:<br />
[address1[,address2]]b[label]<br />
无条件地将控制转移到脚本的其他位置的:label处。也就是说,label后面的命令是应用于当前行的下一个命令。如果没有指定label,<br />
控制将一直到达脚本的末端,因此不再有命令作用于当前行。</p>
<p>5.c<br />
用法:<br />
[address1[,address2]]c<br />
text<br />
用text替代(改变)由地址选定的行。当指定的是一个行范围时,将所有的这些行作为一个组由text的一个副本来替代。每个text行后面的换行符必须用反斜杠将其转义,但最后一行除外。实际上,模式空间的内容被删除,因此后续的命令不能应用于它(或应用于text)</p>
<p>6.d<br />
用法:<br />
[address1[,address2]]d<br />
从模式空间中删除行。因此行没有传递到标准输出。一个新的输入行被读取,并用脚本的第一个命令来编辑。</p>
<p>7.D<br />
用法:<br />
[address1[,address2]]D<br />
删除由命令N创建的多行模式空间中的一部分(直到嵌入的换行符),并且用脚本的第一条命令恢复编辑。如果这个命令使模式空间为空,那么将读取一个新的输入行,和执行了d命令一样。</p>
<p>8.g<br />
用法:<br />
[address1[,address2]]g<br />
将保持空间(参见h或H命令)中的内容复制到模式空间中,并将当前的内容清除。<br />
9.G<br />
用法:<br />
[address1[,address2]]G<br />
将换行符后的保持空间(参见h或H命令)内容追加到模式空间。如果保持空间为空,则将换行符添加到模式空间。<br />
10.h<br />
用法:<br />
[address1[,address2]]h<br />
将模式空间的内容复制到保存空间,即一个特殊的临时缓冲区。保存空间的当前内容被清除。<br />
11.H<br />
[address1],address2]]H<br />
将换行符和模式空间的内容追加到保持空间中,即使保持空间为空,这个命令也追加换行符。<br />
12.i<br />
用法:<br />
[address1]i<br />
text<br />
将text插入到每个和address匹配的行的前面<br />
13.l<br />
用法:<br />
[address1[,address2]]l<br />
列出模式空间的内容,将不可打印的字符表示为ASCII码。长的行被折行。<br />
14.n<br />
用法:<br />
[address1[,address2]]n<br />
读取下一个输入行到模式空间。当前行被送到标准输出。新行成为当前行并递增行计数器。将控制转到n后面的命令,而不是恢复到脚本的顶部。<br />
15.N<br />
用法:<br />
[address1[,address2]]N<br />
将下一个输入行追加到模式空间的内容之后;新添加的行与模式空间的当前内容用换行符分隔(这个命令用于实现跨两行的模式匹配。利用n来匹配嵌入的换行符,则可以实现多行匹配模式)。<br />
16.p<br />
用法:<br />
[address1[,address2]]p<br />
打印所寻址的行。注意这将导致输出的重复,除非默认的输出用”#n”或”-n”命令行选项限制。常用于改变流控制(d,n,b)的命令之前并可能阻止当前行被输出。<br />
17.P<br />
用法:<br />
[address1[,address2]]P<br />
打印由命令N创建的多行模式空间的第一部分(直接嵌入的换行符)。如果没有将N应用于某一行则和p相同。<br />
18.q<br />
用法:<br />
[address]q<br />
当遇到address时退出。寻址的行首先被写到输出(如果没有限制默认输出),包括前面的a或r命令为它追加的文本。<br />
19.r<br />
用法:<br />
[address]r file<br />
读取file的内容并追加到模式空间内容的后面。必须在r和文件名file之间保留一个空格。<br />
20.s<br />
用法:<br />
[address1[,address2]]s/pattern/replacement/[flags]<br />
用replacement代替每个寻址的pattern。如果使用了模式地址,那么模式//表示最后指定的模式地址。可以指定下面的标志:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>n 替代每个寻址的行的第n个/pattern/。N是1到512之间的任意数字,并且默认值为1。
g 替代每个寻址的行的所有/pattern/,而不只是第一个
p 如果替换成功则打印这一行。如果成功进行了多个替换,将打印这个行的多个副本。
w file 如果发生一次替换则将这行写入file。最多可以打开10个不同的file。
</code></pre>
</div>
<p>replacement是一个字符串,用来替换与正则表达式匹配的内容.在replacement部分,只有下列字符有特殊含义:</p>
<p>& 用正则表达式匹配的内容进行替换<br />
n<br />
匹配第n个子串(n是一个数字),这个子串以前在pattern中用”(“和”)”指定.<br />
当在替换部分包含”与”符号(&),反斜杠()和替换命令的定界符时可用转义它们.另外,它用于转义换行符并创建多行replacement字符串.</p>
<p>s/pattern/replacememt/flag<br />
如果flag是数字,那么指定对一行上某个位置的匹配.如果没有数字标志,则替换命令只替换第一个匹配串,因此”1”可以被看作是默认的数字标志.</p>
<p>替换元字符是反斜杠()、与符号(&)和n。</p>
<p>反斜杠一般用于转义其他的元字符,但是它在替换字符串中也用于包含换行符。<br />
例如对于如下的行:<br />
column1(制表符)column2(制表符)column3(制表符)column4<br />
使用如下替换语句:<br />
s/制表符//2<br />
注意,在反斜杠的后面不允许有空格。这个脚本产生下面的结果:<br />
column1(制表符)column2<br />
column3(制表符)column4<br />
“与”符号(&)作为元字符表示模式匹配的范围,不是被匹配的行.例如下面的命令:<br />
s/UNIX/s-2&0/g<br />
可以将输入行:<br />
on the UNIX Operating System.<br />
替换成:<br />
on the s-2UNIXs0 Operating System.<br />
当正则表达式匹配单词的变化时,”与”符号特别有用.它允许指定一个可变的替换字符串.诸如”See Section 1.4”或”See Section 12.9”的引用都应该出现在圆括号中,如”(See Section 12.9)”.正则表达式可以匹配数字的不同组合,所以在替换字符串中可以使用”&”并括起所匹配的内容:<br />
s/See Section [1-9][0-9]<em>.[1-9][0-9]</em>/(&)/<br />
这里”与”符号用于在替换字符串中引用整个匹配内容.<br />
n元字符用于选择被匹配的字符串的任意独立部分,并且在替换字符串中回调它.在sed中转义的圆括号括住正则表达式的任意部分并且保存以备回调.一行最多允许保存9次.例如,当节号出现在交叉引用中时要表示为用粗体:<br />
s/(See Section )([1-9][0-9]<em>.[1-9][0-9]</em>)/1fB2fp/<br />
再来看另外一个例子:<br />
$ cat test1<br />
first:second<br />
one:two<br />
$ sed ‘s/(.<em>):(.</em>)/2:1/ test1<br />
second:first<br />
two:one</p>
<p>21.t<br />
用法:<br />
[address1[,address2]]t[label]<br />
测试在寻址的行范围内是否成功执行了替换,如果是,则转移到有label标志的行(参见b和:)。如果没有给出label,控制将转移到脚本的底部。</p>
<p>22.w<br />
用法:<br />
[address1[,address2]]w file<br />
将模式空间的内容追加到file。这个动作是在遇到命令时发生而不是在输出模式空间内容时发生。必须在w和这个文件名之间保留一个空格。在脚本中可以打开的最大文件数是10。如果文件不存在,这个命令将创建一个文件。如果文件存在,则每次执行脚本时将改写其内容,多重写入命令直接将输出写入到同一个文件并追加到这个文件的末端。</p>
<p>23.x<br />
用法:<br />
[address1[,address2]]x<br />
交换模式空间和保持空间的内容。</p>
<p>24.y<br />
用法:<br />
[address1[,address2]]y/abc/xyz/<br />
按位置将字符串abc中的字符替换成字符串xyz中相应字符。</p>
一种CDN中的动态数据存储方案——UbDP
2009-11-22T00:00:00+08:00
CDN
http://chenlinux.com/2009/11/22/a-dynamic-data-storage-solution-for-cdn-ubdp
<p>上回百度到这篇文章,于是上维普下载了来看,为方便阅读,贴在这里。</p>
<p>原文出处《计算机应用》2008年8月第28卷第8期:</p>
<hr />
<p><strong>一种CDN中的动态数据存储方案——UbDP
</strong>汤迪斌,王劲林,倪宏<br />
(1.中国科学院研究生院,北京100039;2.中国科学院声学研究所国家网络新媒体工程技术研究中心,北京100080)<br />
(<a href="mailto:tangdihin@gmail.com">tangdihin@gmail.com</a>)</p>
<p><strong>摘要:</strong>CDN对于动态Web应用的加速通常采用数据缓存或复制技术。针对论坛、博客服务提供商等为注册用户提供个人信息发布平台的网站,提出了一种基于用户的数据分割方法:将数据按所属注册用户进行分割,分布到离该用户最近的数据库系统中。将数据库UID操作分散到多个数据库系统,消除了单个数据库系统的I/O瓶颈。<br />
<strong>关键词:</strong>内容分发网络;分布式数据库;数据缓存;动态Web应用<br />
<strong>中图分类号:TP3l1.133.1 文献标志码:A</strong></p>
<p>0 引言<br />
尽管网络带宽和服务器性能在不断改善,用户访问迟延过大一直是Web应用的一个重要问题。随着流媒体的出现,迟延问题变得更加严重。导致Internet上内容传送迟延过大的原因主要有两个:1)缺乏对Internet的全局管理;2)有限的网络带宽和服务器资源总是跟不上网络负载增长的脚步。解决这个性能问题的基本办法就是将内容移到离最终用户较近的边缘服务器上。内容分发网络(Content Delivery Network,CDN)正是采用了这个办法,减小了访问迟延,提高了数据传输速率。目前CDN主要应用于静态内容的缓存,如静态页面、图片、音视频文件等。对于动态内容的缓存尚处在研究当中,目前主要有边缘计算(Edge Computing)[5,6]、数据库复制(Database Replication)[7,8]、内容感知数据缓存(Content-Aware Cache,CAC)[9-11]、内容无关数据缓存(Content-Blind Cache,CBC)[12,13]四种研究方向。文献[14]的研究表明复制和缓存方法的选择很大程度上依赖于具体的应用,没有一种技术能很好地适用各种Web应用。对于论坛类网站,内容无关缓存方案性能最佳,而对于Amazon类的电子商务网站,数据库复制才是最好的选择。</p>
<p>论坛和博客服务提供商网站的主要特点是:网站内容由大量注册用户产生,每个用户只能添加、修改和删除属于自己的内容,所以对数据库的写操作(UID、Update、Insert and Delete)来源于处于不同位置的用户。本文针对这类为注册用户提供个人信息发布的网站,提出了一种内容分发网络中基于用户的数据分割方法(User-based Data Partition,UbDP)方案:将数据按所属注册用户进行分割,分布到离该用户最近的数据库系统中。<br />
l 内容缓存和内容复制技术<br />
动态网站的普及,驱使CDN和数据库研究组织都在寻找适用于动态内容的先进缓存技术。文献总结了目前主要的四种动态内容缓存和复制技术方案。<br />
边缘计算就是将生成动态页面的代码如PHP代码、ASP代码分发到合适的边缘服务器(Edge Server,ES)上,数据还是保存在内容提供商的原始服务器(Origin Server,OS)上。Es根据Session(用来标识不同的用户会话)、OS上的数据等信息为每个用户生成动态页面。这类方法的优点是结构简单,不需要对原有CDN结构做出调整。缺点是每个数据库查询都需要到0S上执行,增加了迟延,且0S很可能成为性能瓶颈。<br />
数据库复制技术为了解决边缘计算的迟延和数据库瓶颈问题,将数据库也复制到多个ES上,这样ES就能在本地生成动态页面,较大地减小了用户迟延。但是,为了维护数据的一致性,UID操作需要在所有数据库服务器上执行一遍,这会增加大量额外的网络流量和服务器处理开销。</p>
<p>相对于数据库复制技术在ES中复制整个数据库,CAC方案中ES只缓存数据库的一部分。每次查询数据库时,Es都先检查本地数据库中的数据是否足够应答此次查询请求,这个过程称为查询包含检查。如果本地数据足够,则查询请求在本地执行;否则,请求被转发到0S上,ES会缓存查询结果以备后用。这种方案的主要缺点是查询包含检查的计算复杂度太高。<br />
为了降低查询包含检查的计算复杂度,CBC方案得到了应用:ES中根本不运行数据库软件,而只是单独缓存各个数据库查询的结果。因为并不合并各缓存的查询结果,只有当查询语句完全相同时才算缓存命中,所以查询包含检查的计算复杂度大大降低。<br />
以上四种方案都有一个共同的缺点,即所有的UID操作都需要在OS上执行,然后通过一定的策略更新ES上的缓存。这样,在数据库更新相当频繁的应用中,0S上的数据库势必成为系统的性能瓶颈。本文提出的UbDP很好地将UID操作分散到了多个数据库系统,解决了数据库瓶颈问题。<br />
2 基于用户的数据分割<br />
基于用户的数据分割方案应用于传统的CDN架构之上,增加了分布式数据库系统用于存储动态数据,而不改变CDN中原服务器的功能。<br />
<strong>2.1系统架构</strong></p>
<p><a href="http://photo.blog.sina.com.cn/showpic.html#blogid=62d80b5e0100fpe8&url=http://static12.photo.sina.com.cn/orignal/62d80b5et727ea2f919eb&690"><img title="一种CDN中的动态数据存储方案——UbDP" src="http://static12.photo.sina.com.cn/bmiddle/62d80b5et727ea2f919eb&690" alt="一种CDN中的动态数据存储方案——UbDP" /></a></p>
<p>图1 UbDP系统架构</p>
<p>如图1所示。将CDN中的ES按照所在位置划分为有限的多个逻辑区域Region1、Region2、…RegionN,在每个区域内布置一套数据库系统,该数据库系统可以是单台数据库服务器,也可以是由多台数据库服务器组成的Master、Slave结构。数据库中除了存储用户数据外,还维护了用户和其主数据库(Home Database,HDB)的对应关系。每个区域还有一个数据库查询代理(Query Agent,QA),接收所有查询请求,根据查询请求类型进行不同的数据库操作。<br />
ES接收到用户HTTP请求后,根据需要向QA发送数据库查询请求,QA从自身缓存或后台数据库取得数据返回给ES,ES生成返回页面。<br />
全局负载均衡器(Global Server Load Balancing,GSLB)的目的是在整个网络范围内,将用户请求定向到最近的节点。因此,就近性判断是其主要功能。</p>
<p><strong>2.2数据库查询代理</strong></p>
<p><a href="http://photo.blog.sina.com.cn/showpic.html#blogid=62d80b5e0100fpe8&url=http://static2.photo.sina.com.cn/orignal/62d80b5et78f27edbd2f1&690"><img title="一种CDN中的动态数据存储方案——UbDP" src="http://static2.photo.sina.com.cn/bmiddle/62d80b5et78f27edbd2f1&690" alt="一种CDN中的动态数据存储方案——UbDP" /></a></p>
<p>如图2所示,数据库查询代理QA由查询接口、查询分类器、单库查询缓存和多库查询代理组成。<br />
将数据库查询分为三类:第一类为UID操作;第二类只需要查询一个用户的数据,称为单库查询,包括获取用户信息、获取文章内容等,如:<br />
Q1:SELECT name FROM user WHERE userId = 2<br />
Q2:SELECT title FROM post WHERE id = 3 and userId = 2;<br />
Q1从用户表中获取id为2的用户名;Q2从博客文章表中获取用户2的id为3的文章标题。<br />
第三类查询需要同时获取多个用户的数据,称为多库查询,包括用户积分榜、最新博客文章、博客搜索等,如:<br />
Q3:SELECT userId FROM user ORDER BY score DESC LIMIT 0,10<br />
Q4:SELECT postId FROM blog_index WHERE keyword=”旅游”<br />
Q3获取积分最高的10个用户id;Q4从索引表中搜索包含关键字“旅游”的所有博客文章。<br />
查询分类器接收到非UID查询请求后,根据是否包含“userld=”查询条件判断查询所属分类(如果包含“userld=”则是单库查询,否则为多库查询),将请求转发给单库查询缓存或多库查询代理处理。单库查询缓存采用文献[13]中所述的CBC缓存方法,没有命中缓存时的处理流程如下图3所示,先解析出该请求涉及的用户,然后向同区域内的数据库获取该用户的HDB,将查询请求发往HDB,缓存并返回查询结果。</p>
<p><a href="http://photo.blog.sina.com.cn/showpic.html#blogid=62d80b5e0100fpe8&url=http://static16.photo.sina.com.cn/orignal/62d80b5et78f27fa6784f&690"><img title="一种CDN中的动态数据存储方案——UbDP" src="http://static16.photo.sina.com.cn/bmiddle/62d80b5et78f27fa6784f&690" alt="一种CDN中的动态数据存储方案——UbDP" /></a></p>
<p>多库查询代理内部也有一个CBC缓存模块,没有命中缓存时,将查询请求广播到所有数据库系统,综合各数据库查询结果,缓存并返回最终查询结果。为了提高响应速度,对于搜索类不需要对结果进行排序的多库查询,只需一个数据库返回结果,多库查询代理可以立即将结果返回给ES。<br />
<strong>2.3主数据库选择策略</strong><br />
HDB的选择依赖于用户所在的位置,当用户所在位置发生变化时,系统为他选择的HDB也会变化。<br />
1)设置用户注册或第一次登录时所使用ES所在区域为该用户的主区域(Home Region,HR),HR内的数据库为该用户的HDB;HDB将该用户的HDB更新情况广播给所有数据库;<br />
2)HDB记录所有它负责用户UID操作的来源区域;如果发现对于某个用户,上一个时问片断内来自某区域的UID操作多于本区域,则可以判断该用户暂时或永久迁移到了新区域,设置新区域内的数据库为该用户的HDB;<br />
3)将该用户的最新信息同步到新HDB;此过程中,为了避免冲突,需要将该用户数据写锁定,不允许用户更新:选择用户离线时更新其HDB,可以减少写锁定带来的影响;<br />
4)将该用户HDB更新情况广播到所有数据库。根据CDN的请求路由原则,用户请求总是被转发到最合适的Es。理想情况下,从同一个位置发起的多次请求,都会被转发到离用户最近的同一个ES上,称为理想ES。考虑到网络拥塞和ES的负载情况,大多数同一个位置发起的多次请求都会被转发到理想ES同一区域内的ES上。所以只有当用户所在位置发生大范围变动时,才需要更新HDB。实际应用中用户并不会经常性地变化所在区域,所以HDB变化的可能性很小,为用户提供服务的ES通常会和HDB位于网络迟延较小的同一区域内。由于将数据库的读写操作都移到了离用户较近的边缘服务器上,用户迟延大大减小。<br />
3 性能分析<br />
本文用RUBBoS对UbDP和数据集中存储内容无关缓存系统的性能进行了比较。RUBBoS是一个模拟了slashdot.org的基准测试程序,它主要包含了3个数据库表,分别存储了5000000个用户、6000篇文章和200000条评论。我们使用了RUBBoS的PHP实现来进行我们的试验。我们使用RUBBoS的ClientEmulator模拟用户会话来访问网站,用户的平均思考时间为7s,且符合TPC-W标准。边缘服务器和数据库服务器都是2.8GHzCPU,1GB内存,边缘服务器中使用Apache2.2.3和PHP5.2.4,数据库软件为MySQL5.0.22。用NISTNET来模拟广域网环境,图4中实线连接带宽为90Mbps,时延为10ms,表示网络节点在同一个区域内;虚线连接带宽为50Mbps,时延为100ms,表示网络节点位于不同的区域。试验方法是测量不同负载情况下的用户迟延即用户发出请求到收到回复之间的时间差。</p>
<p><a href="http://photo.blog.sina.com.cn/showpic.html#blogid=62d80b5e0100fpe8&url=http://static8.photo.sina.com.cn/orignal/62d80b5et78f29205e7a7&690"><img title="一种CDN中的动态数据存储方案——UbDP" src="http://static8.photo.sina.com.cn/bmiddle/62d80b5et78f29205e7a7&690" alt="一种CDN中的动态数据存储方案——UbDP" /></a></p>
<p>图5 用户只发表评论的性能结果</p>
<p>当用户只进行发表评论操作,即用户的每个请求都会产生数据库UID操作时,测试结果如图5所示。如果认为大于5000ms的用户等待时间是不可接受的,则采用UbDP后,系统的UID容量增加了约40%。系统容量并没有随着服务器的增加而线性增加,原因是:1)没有增加边缘服务器ES的数量,系统整体性能部分受限于ES的计算能力;2)采用UbDP引入了额外的数据库查询负载,每个UID操作前需要执行一个数据库读操作,导致了负载较低时,UbDP的迟延甚至大于内容无关缓存系统。<br />
当用户进行普通浏览时(10%的用户会发表评论,其他用户只是浏览),测试结果如图6所示。在并发用户会话数较小(140以下)时,采用UbDP后平均用户迟延略小于内容无关缓存系统,这是由于UbDP虽然引入了一次额外的数据库查询,但增加的额外查询是在网络迟延较小的本区域数据库上执行的,且原查询也有约一半的几率在本区域数据库上执行。随着并发用户会话数的增加,UbDP由于额外的数据库查询请求,更早进入满负载状态,用户平均迟延逐渐大于内容无关缓存系统。</p>
<p><a href="http://photo.blog.sina.com.cn/showpic.html#blogid=62d80b5e0100fpe8&url=http://static5.photo.sina.com.cn/orignal/62d80b5et78f29e83c744&690"><img title="一种CDN中的动态数据存储方案——UbDP" src="http://static5.photo.sina.com.cn/bmiddle/62d80b5et78f29e83c744&690" alt="一种CDN中的动态数据存储方案——UbDP" /></a></p>
<p>图6 用户普通浏览的性能结果</p>
<p>为了减少额外数据库查询带来的影响,町以在所有ES上增加一个CBC缓存模块,用来缓存用户和HDB的对应关系,CBC接近90%的缓存命中率较大提高了UbDP系统的总体性能(如图6虚线所示UbDP+ES-CBC)。<br />
4 结语<br />
针对论坛和博客服务提供商之类为注册用户提供个人信息发布平台的网站,本文提出了一种内容分发网络中用户分割的分布式数据库系统,通过将不同用户的数据分布到不同的数据库服务器上,有效地将数据库UID操作分散到了多个数据库上,系统的可扩展性大大提高;其代价是增加了额外的读数据库操作,使得在高负载(并发用户数大于140)时,系统性能略低于内容无关缓存系统,但通过在ES上增加CBC模块可以有效减小这一影响。<br />
对于所有数据库表都直接或间接包含某个外键的应用场合,本文所述数据分割方法都适用。</p>
<p><strong>参考文献:</strong></p>
<p>[5]RABINOVICH M,ZHEN XIAO,AGARWAL A. Computing on the edge: A platform for replicating internet applications[C]//Proceedings of the Eighth International Workshop on Web Content Caching and Distribution.Norwell,MA,USA:Kluwer Academic Publishers,2004:57-77;<br />
[6]RABINOVICH M,ZHEN XIAO,DOUGLIS F,et al. Moving edge side includes to the real edge-the clients[C]//Proceedings of the 4th conference on USENIX Symposium on Internet Technologies and Systems.Berkeley,CA,USA:USENIX Association.2003:12;<br />
[7]CECCHET E.C-JDBC:a middleware framework for database clustering[J].IEEE Data Engineering Bulletin,2004,27(2):19-26;<br />
[8]PLATTNER C,ALONSO G.Ganymed:Scalable replication for transactional Web applications[C]//Proceedings of the 5th ACM/IFIP/USENIX International Conference on Middleware.New York,NY,USA:Springer-Verlag,2004:155-174;<br />
[9]AMIRI K,PARK S,TEWARI R,et al. DBProxy:A dynamic data cache for Web applications[C]//Proceedings of 19th Internation Conference on Data Engineering.Bangalore,India:IEEE Computer Society,2003:821-831;<br />
[10]BORNHD C,ALTINEL M,MOHAN C,et al. Adaptive database caching with DBCache[J].IEEE Data Engineering Bulletin,2004,27(2):11-18.<br />
[11]BUCHHLZ T,HOCHSTATTER I,LINNHOFFPOPIEN C.A profit maximizing distribution strategy for context-aware services[C]//Proceedings of Second IEEE International Workshop on Mobile Commerce and Services(WMCS’05).Los Alamitos,USA:IEEE Computer Society,2005:144-153;<br />
[12]OLSTON C,MANJHI A,GARROD C,et al. A sealability service for dynamic Web applications[C]//Proceedings of the Conference on Innovative Data Systems Research(CIDR).Asilomar,CA,USA:ACM Press,2005:56-69;<br />
[13]SIVASUBRAMANIAN S,PIERRE G,VAN STEEN M,et al. GlobeCBC:Content-blind result caching for dynamic Web applications[R].Amsterdam,The Netherlands:Vrije Universiteit,2006;<br />
[14]SIVASUBRAMANIAN S,PIERRE G,VAN STEEN M,ALONSO G.Analysis of caching and replication strategies for Web applications[J].IEEE Internet Computing,2007,11(1):60-66;<br />
[15]Rice University,INRIA.RUBBoS:Bulletin Board Benchmark[EB/OL].[2008-01-21]. <a href="http://jmb.objeetWeb.org/rnbbos.html">http://jmb.objeetWeb.org/rnbbos.html</a>;<br />
[16]National Institute of Standards and Technology.NIST net home page[EB/OL].[2008-01-21].http://www-x.antd.nist.gov/nistnet/;</p>
nginx编译优化压力测试(转)
2009-11-21T00:00:00+08:00
nginx
http://chenlinux.com/2009/11/21/zz-nginx-compile-optimization-test
<p>默认nginx使用的GCC编译参数是-O,需要更加优化可以使用以下两个参数:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>--with-cc-opt='-O3' --with-cpu-opt=*
</code></pre>
</div>
<p>具体是什么cpu可以grep name /proc/cpuinfo查看,如果是inter xeon,就写–with-cpu-opt=pentium,如果是AMD,就写–with-cpu-opt=opteron。这样编译针对特定CPU以及增加GCC的优化。</p>
<p>针对优化后的结果,我们进行测试,结果表明使用-O2以及以上的参数,可以微量增加性能1%左右,而O2和O3基本可以认为是相同的。<br />
<code class="highlighter-rouge">bash
./http_load -parallel 100 -seconds 10 urls
10811 fetches, 100 max parallel, 5.23252e+06 bytes, in 10 seconds
1.默认参数 -O
1087.2 fetches/sec, 526204 bytes/sec
msecs/connect: 45.5374 mean, 63.984 max, 1.008 min
msecs/first-response: 45.7679 mean, 64.201 max, 2.216 min
1088.9 fetches/sec, 527027 bytes/sec
msecs/connect: 45.0159 mean, 65.291 max, 0.562 min
msecs/first-response: 46.1236 mean, 67.397 max, 9.169 min
1102.2 fetches/sec, 533465 bytes/sec
msecs/connect: 44.5593 mean, 67.649 max, 0.547 min
msecs/first-response: 45.499 mean, 67.849 max, 2.495 min
2.优化编译后 -O2
1081.1 fetches/sec, 523252 bytes/sec
msecs/connect: 45.7144 mean, 63.324 max, 0.823 min
msecs/first-response: 46.1008 mean, 61.814 max, 4.487 min
1110.2 fetches/sec, 537337 bytes/sec
msecs/connect: 43.4943 mean, 60.066 max, 0.715 min
msecs/first-response: 45.756 mean, 62.076 max, 3.536 min
1107 fetches/sec, 535788 bytes/sec
msecs/connect: 44.872 mean, 3036.51 max, 0.609 min
msecs/first-response: 44.8625 mean, 59.831 max, 3.178 min
3.优化编译后 -O3
1097.5 fetches/sec, 531189 bytes/sec
msecs/connect: 45.1355 mean, 3040.24 max, 0.583 min
msecs/first-response: 45.3036 mean, 68.371 max, 4.416 min
1111.6 fetches/sec, 538014 bytes/sec
msecs/connect: 44.2514 mean, 64.831 max, 0.662 min
msecs/first-response: 44.8366 mean, 69.904 max, 3.928 min
1099.4 fetches/sec, 532109 bytes/sec
msecs/connect: 44.7226 mean, 61.445 max, 0.596 min
msecs/first-response: 45.4883 mean, 287.113 max, 3.336 min
</code></p>
squidclient用法
2009-11-21T00:00:00+08:00
squid
http://chenlinux.com/2009/11/21/squidclient-usage
<p>squidclient是squid自带的一个小工具,一般用的最多的,就是-m purge URL了。<br />
其实还有别的用法(看看help吧,不过这个慢慢来~~)先说一个大全式的用法(其他的具体用法,大不了用这里rep出来好了):squidclient -p 80 mgr:info,显示如下:</p>
<p>[root@raocl ~]# /home/squid/bin/squidclient -p 80 mgr:info<br />
HTTP/1.0 200 OK<br />
Server:<br />
squid/2.6.STABLE21<br />
squid版本<br />
Date: Sat, 21 Nov 2009 03:58:45 GMT<br />
Content-Type: text/plain<br />
Expires: Sat, 21 Nov 2009 03:58:45 GMT<br />
Last-Modified: Sat, 21 Nov 2009 03:58:45 GMT<br />
X-Cache: MISS from cdn.21vianet.com<br />
Connection: close<br />
Squid Object Cache: Version 2.6.STABLE21<br />
Start Time: Sat, 21 Nov 2009 03:48:16 GMT<br />
Current Time: Sat, 21 Nov 2009 03:58:45 GMT<br />
Connection information for squid:<br />
Number of clients accessing<br />
cache: 1<br />
当前链接客户端数量<br />
Number of HTTP requests<br />
received: 2<br />
当前链接请求数<br />
Number of ICP messages<br />
received: 0<br />
Number of ICP messages<br />
sent: 0<br />
Number of queued ICP<br />
replies: 0<br />
Number of HTCP messages<br />
received: 0<br />
Number of HTCP messages<br />
sent: 0<br />
Request failure ratio:<br />
0.00<br />
Average HTTP requests per minute since<br />
start: 0.2<br />
每分钟链接请求数<br />
Average ICP messages per minute since<br />
start: 0.0<br />
Select loop called: 118332 times, 5.309 ms<br />
avg<br />
Cache information for squid:<br />
Request Hit Ratios: 5min: 0.0%,<br />
60min:<br />
0.0%<br />
五分钟cache请求数命中率<br />
Byte Hit Ratios: 5min: -0.0%,<br />
60min:<br />
100.0%<br />
五分钟cache字节数命中率<br />
Request Memory Hit Ratios: 5min:<br />
0.0%, 60min: 0.0%<br />
Request Disk Hit Ratios: 5min:<br />
0.0%, 60min: 0.0%<br />
Storage Swap size: 6224<br />
KB cache_dir使用大小<br />
Storage Mem size: 104<br />
KB cache_mem使用大小<br />
Mean Object Size: 4.60 KB<br />
Requests given to<br />
unlinkd: 0<br />
Median Service Times (seconds) 5<br />
min 60<br />
min:<br />
HTTP Requests<br />
(All):<br />
0.00000 0.00000<br />
Cache<br />
Misses:<br />
0.00000 0.00000<br />
Cache<br />
Hits:<br />
0.00000 0.00000<br />
Near<br />
Hits:<br />
0.00000 0.00000<br />
Not-Modified Replies:<br />
0.00000 0.00000<br />
DNS<br />
Lookups:<br />
0.00000 0.00000<br />
ICP<br />
Queries:<br />
0.00000 0.00000<br />
Resource usage for squid:<br />
UP Time: 628.201 seconds<br />
CPU Time: 0.216 seconds<br />
CPU Usage: 0.03%<br />
CPU Usage, 5 minute<br />
avg: 0.00%<br />
CPU Usage, 60 minute<br />
avg: 0.04%<br />
Process Data Segment Size via sbrk(): 3040<br />
KB<br />
Maximum Resident Size: 0 KB<br />
Page faults with physical i/o: 0<br />
Memory usage for squid via mallinfo():<br />
Total space in<br />
arena: 3052<br />
KB<br />
Ordinary<br />
blocks:<br />
3038<br />
KB<br />
7 blks<br />
Small<br />
blocks:<br />
0<br />
KB<br />
0 blks<br />
Holding<br />
blocks:<br />
231572<br />
KB<br />
5 blks<br />
Free Small<br />
blocks:<br />
0 KB<br />
Free Ordinary<br />
blocks:<br />
13 KB<br />
Total in<br />
use:<br />
234610 KB 100%<br />
Total<br />
free:<br />
13 KB 0%<br />
Total<br />
size:<br />
234624 KB<br />
Memory accounted for:<br />
Total<br />
accounted:<br />
571 KB<br />
memPoolAlloc calls: 76381<br />
memPoolFree calls: 71547<br />
File descriptor usage for squid:<br />
Maximum number of file<br />
descriptors:<br />
655360<br />
系统文件描述符个数<br />
Largest file desc currently in<br />
use:<br />
44<br />
使用过的最大个数<br />
Number of file desc currently in<br />
use:<br />
41<br />
目前使用中的个数<br />
Files queued for<br />
open:<br />
0<br />
Available number of file descriptors:<br />
655319<br />
Reserved number of file<br />
descriptors: 100<br />
Store Disk files<br />
open:<br />
0<br />
IO loop<br />
method:<br />
epoll<br />
Internal Data Structures:<br />
1379<br />
StoreEntries cache_dir中的文件个数<br />
26 StoreEntries with<br />
MemObjects<br />
mem中的文件个数<br />
25 Hot Object Cache<br />
Items<br />
热点文件个数<br />
1353 on-disk objects</p>
<p>另外,还有一个squidclient -p 80 mgr:objects,号称是慎用的大杀器,能够列出缓存文件列表——问题是:我找了个新开squid的测试机一试,发现列表显示内容是这个样子的:<br />
KEY 29E6623108A3E8A64DBBD016AB239AEA<br />
STORE_OK<br />
NOT_IN_MEMORY SWAPOUT_DONE<br />
PING_NONE<br />
CACHABLE,DISPATCHED,VALIDATED<br />
LV:1258460822 LU:1258461412 LM:1258460823<br />
EX:1260950400<br />
0 locks, 0 clients, 1 refs<br />
Swap Dir 0, File 0X000412<br />
上头是32位MD5值么?谁看的懂呢……</p>
让squid访问日志显示完整url
2009-11-21T00:00:00+08:00
squid
http://chenlinux.com/2009/11/21/show-full-path-in-squid-access-log
<p>一大串的header结束了,从上篇开始回到squid.conf本身的设置上来。</p>
<p>这里说一个小东西,一般用不上,但难保哪天就用了:</p>
<p>我们在access.log里,常看到很多url只显示到?,之后的东西就都忽略掉了,这一是为了醒目,反正都不缓存,显示干嘛;二是为了保密,说不定有些网站的GET就直接把什么秘密信息都明文传输呢?</p>
<p>但有时候特殊情况要求我们调试squid,比方说之前提到过的非得缓存?,却又没有提前告知,临时想要自己找完整url,怎么办?</p>
<p>其实有办法,squid配置中有一个strip_query_terms,就是管这事儿的。默认是on,只要改成off,就可以了~~嘿嘿,干坏事的机会到来了……</p>
cache驻留时间(四、If-Modified-Since)
2009-11-21T00:00:00+08:00
CDN
cache
http://chenlinux.com/2009/11/21/intro-if-modified-since
<p>话接上回If-Modified-Since,当squid开启reload_into_ims on之后,no-cache头会在在浏览器上被转化成If-Modified-Since标识返回给web服务器。从整体架构考虑,因为squid上已经破坏了http协议的规定,那么web端就必须主动承担对网页过期的识别管理工作。嗯,要是所有的网站都能从一规划开始就这么搞,俺们干CDN的可就轻松了~~~</p>
<p>下面是一段php代码,简单的实现对If-Modified-Since标签的过期管理:<br />
<code class="highlighter-rouge">php
<?php
$headers = apache_request_headers();
//读取整个header信息
$client_time = (isset($headers['If-Modified-Since']) ?
strtotime($headers['If-Modified-Since']) : 0);
//判断header信息是否包含If-Modified-Since标签,有则转换其时间为Unix格式,否则退出这段定义
$now=gmmktime();
//web服务器的系统时间,为处理方便转换为GMT
$now_list=gmmktime()-60*5;
//五分钟前的时间
if ($client_time<$now and $client_time
>$now_list){
//判断浏览器时间是不是在当前的五分钟内
header(’Last-Modified: ‘.gmdate(’D, d M Y H:i:s’,
$client_time).’ GMT’, true, 304);
exit(0);
//判断为真,则给header加上时间为浏览器时间的Last-Modified标签,告知浏览器网页未过期
}else{
header(’Last-Modified: ‘.gmdate(’D, d M Y H:i:s’, $now).’ GMT’,
true, 200);
//否则给header加上时间为服务器系统时间的Last-Modified标签,告知浏览器网页过期,重新下载
}
?>
</code><br />
这做一个范例,如果用其他的标签定义来控制过期,照葫芦画瓢就行了。比如用Expires控制,就写</p>
<div class="highlighter-rouge"><pre class="highlight"><code>header('Expires: ' . gmdate ("D, d M Y H:i:s", gmmktime() + 60*5). " GMT");
</code></pre>
</div>
<p>题外话一句,php中关于date的函数很多,各种的格式不同,小心使用。好比这个新浪博客,就有一个小问题,如果你半夜写博客,过了零点以后发表,会提示错误;甚至如果你原先是10点发表的,隔了几天半月的哪天下午14点来修改保存,也会提示错误。非得要你改成当前分钟之前才行……</p>
cache驻留时间(五、Etag)
2009-11-21T00:00:00+08:00
CDN
cache
http://chenlinux.com/2009/11/21/intro-etag
<p>浏览器的请求中,除了用If-Modified-Since去比对Last-Modified以外,还有另一个标签Etag,这个东东是web服务器比较有用的,squid倒没什么。不过大文件下载加速也有采用apache的系统,一并算在cache缓存里头讲吧:</p>
<p>Etag是为了更好的区分文件过期的标签。比如Last-Modified吧,如果我在一秒钟内更新两次这个文件,他的Last-Modified时间还是一样的。但Etag值不一样。听起来很浪费的样子,呵呵~~</p>
<p>Etag是由文件的inode、mtime、size三个数据通过计算得出来的字符串。这三个都可以通过stat命令查看得到。但问题就来了——一个LVS下挂了七八台RealServer,size固然得一样,mtime只要同步没有问题,也是一样,可这个inode,几乎就不太可能一样了——也就是说,当浏览器的请求被LVS转发到A服务器上时,取得了一个Etag值,一旦刷新重复请求,可能LVS又转发到B服务器,而这个文件在B服务器上计算出的Etag值是另一个。浏览器将判断文件不一致,重新下载!</p>
<p>这种情况,于网民,是用户体验度下降;于网站,是带宽重复占用;于服务器,是无效负载……</p>
<p>不过我们既然知道了原理,解决起来也很简单,只需要在apache的配置中,规定Etag的计算来源就行了。</p>
<p>默认的Etag计算是:FileETag INode MTime Size,如果改全局,只要改成FileETag MTime Size就可以了。如果是下面的某一部分,则写成FileETag -INode也行。<br />
header信息就说到这里吧,最后提供一个<a href="http://en.wikipedia.org/wiki/List_of_HTTP_headers">wiki</a></p>
cache驻留时间(六、大文件)
2009-11-21T00:00:00+08:00
CDN
cache
http://chenlinux.com/2009/11/21/intro-cache-object-size
<p>话说上回提到大文件下载,公司除了apache以外,有些也是用squid做的。这回说说这方面的设置。<br />
下载业务,首先要注意到的,第一是多线程,第二是断点续传。</p>
<p>第一个参数:maximum_object_size——这个参数规定了squid能缓存的最大文件大小;<br />
第二个参数:range_offset_limit——这个参数规定了squid能预先读取的文件大小;<br />
第三个参数:quick_abort_(min|max|pct)——这几个参数规定了squid是否继续传输中断请求的文件;</p>
<p>第一个参数很熟悉,就不用说了。</p>
<p>第二个参数,squid的官方解释是:Sets a upper limit on how far into the the file a Range request may be to cause Squid to prefetch the whole file.</p>
<p>也就是说这个参数当客户端请求的header中带有range标签时(也就是多线程下载),如果文件大小在这个参数的规定范围内,squid会预读取这段文件作为缓存。<br />
但是要注意了:如果你把range_offset_limit设的比maximum_object_size还大的话,squid按规则,会每次预读取完文件之后,再毫不犹豫的把它从cache里扔出去!!</p>
<p>这还不算完…如果网民这个线程开的比较猛,并发上20个线程来下载,对于squid,它却只认其中一个最快的一个线程,也就是说很有可能在第一次缓存的时候,真正下载的流量,是文件大小的20倍,于是很大可能又超过了maximum_object_size,squid又毫不犹豫抛弃掉……带宽不是用来这么玩的呀~~其结果我们也看得到,那就是cacti上上窜下跳的流量图。</p>
<p>第三个参数,当客户端中断请求后,squid会比对文件剩余部分的大小,如果小于min,就继续从源站下载;如果大于max,就放弃;如果达到了pct的比率,也继续——嗯,很像refresh_pattern的定义模式。</p>
<p>【基本上就是这些,另外,和maximum_object_size相关的还有一个maximum_object_size_in_memory,也就是能缓存在内存里的大小。这是另一个方向,要是够狠,完全可以把squid全跑在mem里(–enable-storeio)把cache_dir设成null……】</p>
<p>最后,如果修改了这些参数,对普通的小文件加速服务,会有一定的冲击,最好的办法,还是在后端的web架构上进行区分;其次,把squid进行区分,一部分专门跑下载,而其他的禁用掉range,向跑下载的邻居转发请求。不过sibling靠ICP获取一个列表的摘要,很可能假命中。这又需要对下载邻居进行详细限定,架构变得复杂无比。。。</p>
cache驻留时间(三、Expires/Cache-Control)
2009-11-20T00:00:00+08:00
CDN
cache
http://chenlinux.com/2009/11/20/intro-expires-cache-control
<p>在谈cache的时候插入了上一篇回忆正则表达式的内容,是因为最近一个客户的古怪要求“url以/结尾或文件夹结尾的不能缓存”。</p>
<p>惯常的要求,大多有.*/$的缓存或者不缓存,这个文件夹不带/的结尾,可就没见过了。</p>
<p>思前想后,我觉得大概文件夹和文件的区别就是没有后缀名了吧。于是回忆了好一番正则表达式,最后写成下面那么一句:<br />
<code class="highlighter-rouge">squid
acl test url_regex -i ^<a href="http://www.test.com/.*/[^.]*">http://www.test.com/.*/[^.]*</a>$
cache deny test
</code><br />
测试访问结果都是MISS/200——可惜还没高兴起来呢,赫然发现其header里写着:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-ridate, post-check=0,
pre-check=0
Pragma: no-cache
</code></pre>
</div>
<p>敢情MISS不是我的acl起作用了,是人家web端早就定义好了……<br />
转过头来,继续研究cache和header的关系。今天就看这个Expires、Cache-Control和Pragma:</p>
<ul>
<li>
<p>Expires 申明文件的过期时间,比如这里申明的是1981年——默认不缓存的设置一般就是1981年;</p>
</li>
<li>
<p>Cache-Control 常见设定和说明见下图:<br />
<img src="/images/uploads/cache-control.jpg" alt="" /><br />
而且对于网民本地的浏览器来说,不同的操作,导致的结果也不同;</p>
</li>
<li>
<p>Pragma<br />
这个的说明不好找,大约是在HTTP1.0里使用的一种定义,我就见过no-cache一个选项,而且在HTTP1.1里,强烈警告不要使用这个标签。</p>
</li>
</ul>
<p>概念都复述完了,然后是办法:squid可以在refresh_pattern段里使用各种选项把这些header一一忽略,当然,这些个个都是有违http精神的做法~~除了上期提到的ignore-reload以外,还有:ignore-no-cache ignore-private ignore-no-store ignore-auth ignore-revalidate(这些对2.6都要打补丁,更高版本集成了)。<br />
这样的处理以后,关于客户端的网页缓存控制,就由web服务器对浏览器的If-Modified-Since进行判别控制了,可以说,除非是web方面有经验的做这方面的工作,不然的话,CDN上出错的概率很大……<br />
最后提一句:文前提到的这个客户,后来我居然发现他有个文件夹目录是http://www.test.com/update/v2.4,无语了,天知道还有什么别的稀奇古怪的目录命名……</p>
awk单行实践
2009-11-19T00:00:00+08:00
bash
awk
http://chenlinux.com/2009/11/19/use-awk-as-one-line-command
<p>客户提交一份预加载文件列表,采用了如下格式:<br />
http://www.a.com/a/b/<br />
a b c d e f g<br />
h i j k l m n<br />
http://www.b.com/b/c/<br />
o p q r s t<br />
u v w x y z<br />
http://www.c.com/d/e<br />
http://www.d.com/f/g<br />
必须要把文件整理成完整的url,才好操作。<br />
最初的设想,是以带http开头的行为RS,以n为OFS,然后打印RS<br />
$0。随后发现这个想法问题多多——最主要的一点是:直接print $0的话,输出结果是不显示OFS的。<br />
然后我才想到用for循环打印所有列的话,默认就已经分行了,不用定义OFS和ORS。<br />
剩下的问题就是RS,然而不管我怎么写正则匹配表达式,结果都搞不定……唉<br />
最后只能放弃这个想法,采用比较繁琐的办法:</p>
<p>awk -v RS=”http” ‘{if($2==””){print RS$1>url}else{for(i=2;i<=NF;i++){print RS$1$i>url}}}’ URLFILE</p>
<p>需要注意的一点,这里RS$1$i之间有没有空格(但不能是逗号)都不影响结果,但如果是{x=$1}{print RS x$i}的话,RS和x之间就必须有空格!<br />
这样5000个错乱的url,一敲回车就输出成一个url文件,列好了完整的url列表。然后for;do wget;done搞定预加载~~</p>
正则表达式
2009-11-19T00:00:00+08:00
http://chenlinux.com/2009/11/19/regular-expression
<p>系统整理一下正则表达式:<br />
“.”匹配除了”n”以外的任一字符;<br />
“<em>”匹配前一字符任意次;【“.</em>”匹配任意个字符】<br />
“^”匹配行首;<br />
“$”匹配行尾;【“^$”匹配空行】<br />
“[ ]”匹配其中某个字符;【“[^ ]”匹配此外任一字符】<br />
“”匹配单词边界,即完整单词;<br />
“?”匹配前一字符0或1次;<br />
“+”匹配前一字符非0次;【可能要用转义一下】<br />
“{ }”匹配次数;</p>
cache驻留时间(二、LM-factor算法)
2009-11-19T00:00:00+08:00
CDN
cache
http://chenlinux.com/2009/11/19/intro-lm-factor-algorithm
<p>好吧,满足某人的好奇,花开两朵,各表一枝了。<br />
今天又看到关于LM-factor的另一种说法,特摘录如下:<br />
<img src="/images/uploads/lm-factor.gif" alt="" /><br />
上面这张图来自于《Squid.Definitive.Guide》第七章,对squid的LM-factor算法作出了一个很直观的描述。<br />
请注意这张图的起始时间坐标:Last-Modified,这个是由squid读取的原始web数据所规定的。<br />
然后就是Date,这个是原始数据进入squid的缓冲的时间。<br />
最后就是Expires,这个就是原始数据在squid中的缓冲过期时间。<br />
可以很容易的得出结论,对于LM-factor算法来说,原始数据在squid中的缓冲时间为<br />
(原始数据进入squid的缓冲的时间-原始web数据所规定的Last-Modified时间)<em>percent<br />
所以,我们可以郑重得出结论,在squid的refresh_pattern设置中,percent与Min、Max两个值是完全没有关系!<br />
最后总结一下,对于squid来说,缓冲的数据在cache中的存活时间是这样决定的:<br />
如果有定义refresh_pattern:只要满足以下两个条件之一,缓冲对象过期<br />
缓冲对象在squid的cache缓冲的时间大于refresh_pattern定义的max<br />
缓冲对象在squid的cache缓冲的时间大于(原始数据进入squid的缓冲的时间-原始web数据所规定的Last-Modified时间)</em>percent<br />
用编程语言来描述,就是<br />
if<br />
((CURRENT_DATE-DATE)<br />
elif<br />
((CURRENT_DATE-DATE)/(DATE-LM_DATE)<br />
elif<br />
((CURRENT_DATE-DATE)>max){STABLE}<br />
else{STABLE}</p>
cache驻留时间(一、refresh_pattern)
2009-11-18T00:00:00+08:00
CDN
cache
http://chenlinux.com/2009/11/18/intro-refresh_pattern
<p>上一篇举了个缓存时间的事故,现在说说普通情况下影响这个的配置:</p>
<p>首先,refresh_pattern规则仅仅应用到没有明确过时期限的响应。原始服务器能使用Expires头部,或者Cache-Control:max-age指令来指定过时期限。</p>
<p>然后,介绍一下LM-factor算法:</p>
<p>LM-factor=(response age)/(resource age)</p>
<div class="highlighter-rouge"><pre class="highlight"><code>响应年龄(即response age)=当前时间-对象进入cache的时间
源年龄(即resource age)=对象进入cache的时间-对象的last_modified
</code></pre>
</div>
<p>就好比上篇,因为Last-Modified错误,源年龄变成了三年,而响应年龄相对三年来说太短了,所以LM-factor也就很小很小,永远达不到警戒线……</p>
<p>最后常用刷新选项:</p>
<p>** ignore-reload<br />
该选项导致squid忽略请求里的任何no-cache指令。如果希望内容一进入cache就不删除,直到被主动purge掉为止,可以加上ignore-reload选项,这个我们常用在mp3,wma,wmv,gif之类。</p>
<p>** override-expire<br />
该选项导致squid在检查Expires头部之前,先检查min值。这样,一个非零的min时间让squid返回一个未确认的cache命中,即使该响应准备过期。</p>
<p>** override-lastmod<br />
该选项导致squid在检查LM-factor百分比之前先检查min值。</p>
<p>** reload-into-ims<br />
该选项导致squid以no-cache指令传送确认请求。换句话说,squid在转发请求之前,对该请求增加一个If-Modified-Since头部。注意这点仅仅在目标有Last-Modified时间戳时才能工作。外面进来的请求保留no-cache指令,以便它到达原始服务器。<br />
一般情况可以使用reload-into-ims。它其实是强行控制对象的超时时间,这违反了http协议的精神,但是在带宽较窄的场合,可以提高明显系统相应时间。<br />
以上这段是网上百度的,其实最后一段是废话,squid缓存选项,各个都是有违http协议精神的……</p>
游戏CDN加速猜想
2009-11-18T00:00:00+08:00
CDN
http://chenlinux.com/2009/11/18/guess-the-cdn-accel-for-games
<p>没事去国内几个主要的CDN商网站看了看各个的CDN产品说明和解决方案,基本上大同小异(其实连小异我都没怎么找到)。唯一让我惊讶的是蓝汛Chinacache——蓝汛在自己网站上写着“游戏应用加速”,内容如下:</p>
<p>游戏运营商只需要从一点接入ChinaCache的CDN网络,通过ChinaCache的应用加速服务,不论玩家在国内或者国外,无论何时何地,从任何电信运营商接入,玩家的客户端都能够与游戏运营商的服务器之间高速传递游戏数据,从而保证中国南北甚至全球玩家都可以畅快淋漓的参与即时游戏;</p>
<p><img src="http://www.chinacache.com/uploadfile/20080110170409857.jpg" alt="" /></p>
<p>图中的CCR,百度了一下,好像是蓝汛的自动调整流量导向的监控管理平台。</p>
<p>这个图画的不明不白的,对我理解其加速实在是有害无益——话说回来,似乎所有公司的公开解决方案都是这么云里雾里——不如自己去假设:</p>
<p>网游的架构,大体应该是客户端的动作,按照代码生成一个.dat临时文件,并随时上传到服务器端;服务器端定时以及当客户端主动保存时,将该临时文件的数据记录进数据库中存储;等到客户端重新读档(或者掉线回档)时,从数据库中读取最近一次保存的数据。</p>
<p>【以上说法来自我的室友,也是我公司游戏部的同事】</p>
<p>这是上下两个过程,对于往下走的读档流程来说,实在没什么太大的加速价值,因为它是个一锤子买卖(除非是搞成分布式数据库,但这跟CDN什么关系?);对于往上走的存档流程来说,倒是有些地方可以想想?</p>
<p>关键就在这个.dat临时文件上。我们完全可以让cache服务器缓存这些文件,然后定时上传到上级节点或者源站;还需要开发一个触发器,以便客户端主动存档使用。</p>
<p>不过这些想法,都是建立在尽量传统的CDN理解即内容分发上。而蓝汛一向号称自己有独一无二的动态应用加速手段,从网上找到的GAD白皮书来看,核心还是路由优化而已。恐怕游戏应用加速,也逃脱不了这个范围。<br />
在边写这篇笔记边百度的时候,发现一篇论文《一种CDN中的动态数据存储方案——UbDP》,打算想办法下载全文来看看,其摘要如下:“<br />
CDN对于动态Web应用的加速通常采用数据缓存或复制技术。针对论坛、博客服务提供商等为注册用户提供个人信息发布平台的网站,提出了一种基于用户的数据分割方法:将数据按所属注册用户进行分割,分布到离该用户最近的数据库系统中。将数据库UID操作分散到多个数据库系统,消除了单个数据库系统的I/O瓶颈。<br />
”<br />
相信不会让人失望。</p>
squid一次诡异事故
2009-11-18T00:00:00+08:00
squid
http://chenlinux.com/2009/11/18/a-strange-accident-of-squid
<p>前几天出了一次诡异的事情。某客户在半夜2点钟更新了其网站的内容后,按照刷新规则,squid应该在15分钟内也更新成新内容的。但实际情况却是新网页一刷新没准就变成旧的,一直到5点左右这种现象才算是消失了。</p>
<p>用前段时间写的age测试脚本检查全部服务器,赫然发现有些服务器上的测试文件的age卡在102164686,也就是三年多!</p>
<p>经过整整一个下午的刷新观察,所有的服务器都陆续出现过这种情况,然后不定什么时间age又突然回复正常计数一段时间…等了很久,捕捉到一个现象,就是金华节点的测试age在896的时候,我一刷新,变成102164686了。也就可以认为,这个服务器的age计数在到达15分钟去源站比对文件的时候,突然变成102164686了。</p>
<p>因为之前脚本过滤了其他信息,只显示HTTP1/0|Age|Cache三行。于是改手动wget看全部信息。结果无意的刷新几遍后,赫然发现有一次header里的Date居然是2006年!难道是这台机器有问题?确认本机date无误后,我又登陆其他节点几台机器一一试验,都出现这个情况……于是在crontab里执行每5分钟从源站wget一次测试文件,过两天来看看结果,如下所示:<br />
<code class="highlighter-rouge">bash
[root@squid1 ~]# cat /root/wget.log|awk '/Last/{print $0}' |sort -n |uniq -c
803 Last-Modified: Thu, 12 Nov 2009 05:58:12 GMT
1 Last-Modified: Thu, 24 Aug 2006 00:09:51 GMT
1 Last-Modified: Thu, 24 Aug 2006 00:24:52 GMT
1 Last-Modified: Thu, 24 Aug 2006 00:29:51 GMT
1 Last-Modified: Thu, 24 Aug 2006 00:44:51 GMT
1 Last-Modified: Thu, 24 Aug 2006 00:49:51 GMT
[root@squid1 ~]# cat /root/wget.log|awk '/Last/{print $5}' |sort -n |uniq -c
381 2006
803 2009
</code><br />
源站文件的Last-Modified时间居然在变化!而且除了正确的2009年时间不变外,2006年的时间居然是随着时间走的(crontab是5分钟,wget日志里每次Last-Modified的时间也是隔5分钟)……</p>
<p>由此基本确定是客户源站的问题,我的理解是:当cache服务器到时去源站比对时间时,如果碰上源站这会儿时间是2009年,就更新文件并重计age;如果碰上源站这会儿时间是2006年了,那cache比源站还新,自然没法变动了……</p>
<p>但让我不解的是:header信息里Date字段时间变化,还可以说是服务器系统时间不正常,文件的Last-Modified为什么会这样变化呢?!</p>
squid的一点小问题
2009-11-17T00:00:00+08:00
squid
http://chenlinux.com/2009/11/17/a-error-page-problem-of-squid
<p>网站运行,出错是必然的。squid提供了一整套多国语言的错误信息页面,放在share/errors/目录下。<br />
但是,让人很尴尬的一点是:squid默认的english错误页面中,居然会公开显示客户源站的IP地址。而有一部分客户,用CDN的目的之一就是要用CDN来分担攻击流量,保护自己。这下可好。生生给暴露出去了。<br />
而附带的简体中文页面中,刚好就没这个信息。真不知道是在讽刺国人攻击性太强,心理太黑暗了么……<br />
不管怎么说,得把这个改掉。最简单的办法,修改english页面,删除掉ERR_CONNECT_FAIL里那个关键的信息。关键的就是下面这一段:</p>
<p>Connection to %I Failed</p>
<p>这个%I,就是源站IP。修改成Connection Failed。显示结果就OK了。<br />
这一次成了,难保下一次别的错误信息里又出什么别的问题。干脆的办法,把错误信息定位到中文包上去。<br />
网上办法多多。</p>
<p>最根本的,在源代码编译的时候,就加上–enable-err-language=Simplify_Chinese;</p>
<p>最简单的,删除掉English目录,创建一个同名链接链接到Simplify_Chinese目录上;</p>
<p>最实用的,在squid.conf里加上一句配置语句“error_directory /home/squid/share/errors/Simplify_Chinese”,重启服务,即可。</p>
squid的SSL配置
2009-11-16T00:00:00+08:00
squid
http://chenlinux.com/2009/11/16/configure-ssl-support-of-squid
<p>公司新进客户,要求加速它的论坛,比较奇怪的是,整个论坛居然都是https协议的网页。所以得做443端口的配置。<br />
如果只是端口,一个https_port 443就够了。麻烦的地方在证书(之前就有客户死活不肯给证书,于是只能给做个端口转发,顶天了算是路由优化,何苦往CDN里投钱……)。<br />
在拿到证书后,squid.conf里添加这么一句,SSL配置就算是完成了。但测试的时候问题可就多多了<br />
<code class="highlighter-rouge">squid
https_port 443 cert=/test_ssl/server.cer key=/test_ssl/server.key defaultsite=bbs.test.com
</code><br />
客户源站在江苏电信,之前的普通静态加速时,为了更好的达到回源效果,所有的网通节点服务器都采用了二级代理的方式,通过BGP回源。 <br />
在按照普通域名走父方式配置完毕后,wget该客户的论坛首页做测试,所有的网通节点返回的状态码倒是200,可首页文件总字节数,只有209!用IE打开一看,赫然是一个“Hello<br />
World!”暴汗…… <br />
改为直接回源后,立马恢复正常。实在不知道这个你好页面是哪里蹦出来的! <br />
把这个问题交给百哥和谷婶,大家伙都说只要加上一条never_direct allow all就可以强制转发https协议到上级cache服务器就可以了。但我测试的结果很遗憾——209依然——不可行!</p>
<p>和同事讨论没什么办法,大家认为这个应该是父节点要采用的证书应该是另外一种,因为他既要接受子的请求,又要去源站发起请求。目前情况下,只能取消走父配置,统一直接回源而已。然后缓存,刷新时间等等测试一一通过。唉~<br />
[root@squid1 ~]# wget -S -O /dev/null https://bbs.test.com/attachments/month_0910/20091014_c30caa02ae844a8dbe58M7PIVoPJPjMh.jpg –no-check-certificate<br />
–20:20:28–<br />
https://bbs.test.com/attachments/month_0910/20091014_c30caa02ae844a8dbe58M7PIVoPJPjMh.jpg<br />
Resolving bbs.test.com… 1.2.3.4<br />
Connecting to bbs.test.com|1.2.3.4|:443… connected.<br />
WARNING: cannot verify bbs.test.com’s certificate, issued by<br />
<code class="highlighter-rouge">/C=BE/OU=Domain Validation CA/O=GlobalSign nv-sa/CN=GlobalSign Domain Validation CA':
Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response...
HTTP/1.0 200 OK
Date: Mon, 16 Nov 2009 09:02:07 GMT
Server: Apache/2.2.9 (Unix) DAV/2 mod_ssl/2.2.9 OpenSSL/0.9.8h PHP/5.2.6 mod_apreq2-20051231/2.6.0 mod_perl/2.0.4 Perl/v5.10.0
Last-Modified: Wed, 14 Oct 2009 13:52:59 GMT
ETag: "efc217-1801e-475e57b0924c0"
Accept-Ranges: bytes
Content-Length: 98334
Content-Type: image/jpeg
Age: 1
X-Cache: HIT from cdn.21vianet.com
Connection: keep-alive
Length: 98334 (96K) [image/jpeg]
Saving to: </code>/dev/null’<br />
100%[===================================================================================================================>]<br />
98,334<br />
356K/s in<br />
0.3s<br />
20:20:34 (356 KB/s) - `/dev/null’ saved [98334/98334]</p>
<p>这里要注意两个地方。</p>
<p>第一,刷新配置的匹配字段不再是^http://bbs.test.com/.<em>而是^https://bbs.test.com/.</em>,不然的话,age值就按之后的默认配置计数了; <br />
第二,wget测试的时候,必须使用“–no-check-certificate”参数,否则没法测试;同理,如果是用curl测试,也必须加上“-k”或者“–insecure”参数。</p>
<p>curl的结果类似下面这样:</p>
<p>[root@squid1 ~]# curl -I https://bbs.test.com/attachments/month_0910/20091014_c30caa02ae844a8dbe58M7PIVoPJPjMh.jpg -k<br />
HTTP/1.0 200 OK<br />
Date: Mon, 16 Nov 2009 09:02:07 GMT<br />
Server: Apache/2.2.9 (Unix) DAV/2 mod_ssl/2.2.9 OpenSSL/0.9.8h<br />
PHP/5.2.6 mod_apreq2-20051231/2.6.0 mod_perl/2.0.4<br />
Perl/v5.10.0<br />
Last-Modified: Wed, 14 Oct 2009 13:52:59 GMT<br />
ETag: “efc217-1801e-475e57b0924c0”<br />
Accept-Ranges: bytes<br />
Content-Length: 98334<br />
Content-Type: image/jpeg<br />
Age: 187<br />
X-Cache: HIT from cdn.21vianet.com<br />
Connection: close</p>
squid防盗链配置
2009-11-14T00:00:00+08:00
squid
http://chenlinux.com/2009/11/14/anti-hotlinking-in-squid
<p>做网站的,谁愿意自己辛辛苦苦的成果就被别人轻松转载,如果是文字的,一般也就禁鼠标右键,再没什么好办法(当然,名人好打官司另说),但如果是图片,影音的文件,大可以利用http协议的header信息进行控制,这就是大多数web服务器日志要记录的referer。<br />
公司新进一测试客户,就要求CDN方配合做防盗链。</p>
<p>公司自然有规范,直接ctrl+c、ctrl+v就搞定。但这些句子,还是值得细细研究一下的。<br />
相关语句如下:<br />
<code class="highlighter-rouge">squid
acl test_domain dstdomain .test.com
acl null_referer referer_regex .
acl right_referer referer_regex -i
^http://test.com ^http://.*.test.com
http_access allow test_domain !null_referer
http_access deny test_domain !right_referer
</code><br />
第一关键点,是第一行的那个“.”,“.”匹配的是除了“n”以外的任何一个字符。那么!null_referer也就是“n”,也就是说第一条access定义的,是允许referer为空行;</p>
<p>第二关键点,是access的“!”,“!”就是非,那么!right_referer定义的就是一切除了test.com以外的域名,也就是说第二条access定义的,是不允许所有其他网站。</p>
<p>这样的结果,也就是只有从test自己的网站,或者直接在浏览器地址栏里输入完整url,才能看到文件(linux上常用的wget、curl,默认的referer也是空,所以也可以。我又试试迅雷,其referer也是空,那么估计下载工具也都是这样)</p>
<p>(比较奇怪的一点是:squid的日志里,空不显示为“ ”,而是“-”,很能迷惑人呀!)</p>
<p>于是我想到新浪和百度呀这些博客之间转来转去的图片,一般都显示一个空图,但点开来(或许还要再刷一次)也一样能看。可见防盗链都是这么做的。</p>
<p>如果真就狠到了连直接url查看也不让,那就把null_referer的定义删除掉,自然也就可以了……</p>
<p>试到这里,发现另一个问题:nagios的监控,一般也是空referer的,如果真这么狠的要求,这个监控也得改了。<br />
因为不管是curl还是wget,都可以伪装referer。<br />
两个的伪装语法分别是:<br />
curl -e “http://www.test.com” -x $squidip:80 http://www.test.com/test.gif<br />
wget http://www.test.com/test.gif –refer=”http://www.test.com” -e “http_proxy=$squidip”</p>
<p>我对nagios不熟,不知道里面具体是用什么去check的,大概也差不离吧?<br />
最后,像新浪百度这样的盗链显示图片怎么做的?也就是一句话的事,如下:<br />
<code class="highlighter-rouge">squid
deny_info http://www.test.com/你盗链啦.gif
right_referer
</code></p>
awk变量(三续)
2009-11-12T00:00:00+08:00
bash
awk
http://chenlinux.com/2009/11/12/awk-variable-3
<table>
<tbody>
<tr>
<td>网上闲逛,偶然看到一句统计TCP连接数的命令如右:netstat -n</td>
<td>awk ‘/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}’</td>
</tr>
</tbody>
</table>
<p>要统计TCP连接数,其实用上wc命令,倒不甚难。不过为了熟悉NF的用途,便细细试试这条吧。<br />
先看试验中netstat -n的结果:<br />
<code class="highlighter-rouge">bash
[root@raocl rao]# netstat -n |awk '/^tcp/{print $0}'
tcp 0 0 211.151.70.76:80 60.12.137.170:3157 SYN_RECV
tcp 0 0 211.151.70.76:80 117.136.0.184:53476 SYN_RECV
tcp 0 0 211.151.70.76:80 112.193.8.170:4281 SYN_RECV
tcp 0 0 211.151.70.76:80 112.65.48.252:62480 ESTABLISHED
tcp 0 0 211.151.70.76:80 113.205.102.168:3230 ESTABLISHED
tcp 0 0 211.151.70.76:80 198.54.202.250:2714 ESTABLISHED
tcp 0 1370 211.151.70.76:80 222.44.43.141:2070 FIN_WAIT1
tcp 0 0 211.151.70.76:80 220.248.86.74:53112 ESTABLISHED
……(下略)
</code><br />
然后详细打印一下那条命令里NF的每一个变化使用值:<br />
<code class="highlighter-rouge">bash
[root@raocl rao]# netstat -n | awk '/^tcp/ {print ++S[$NF],S[$NF],$NF,NF}'
……(上略)
532 532 ESTABLISHED 6
8 8 LAST_ACK 6
533 533 ESTABLISHED 6
534 534 ESTABLISHED 6
33 33 TIME_WAIT 6
535 535 ESTABLISHED 6
536 536 ESTABLISHED 6
</code><br />
也就是利用awk的行处理特性,遍历了所有tcp开头的行。定义出不同状态命名的数组下标,并分别++计数赋值给数组元素。<br />
最后,打印数组S,如下:<br />
<code class="highlighter-rouge">bash
[root@tinysquid2 ~]# netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
TIME_WAIT 18
FIN_WAIT1 33
FIN_WAIT2 1
ESTABLISHED 508
SYN_RECV 5
LAST_ACK 11
</code></p>
正则表达式一例
2009-11-12T00:00:00+08:00
squid
http://chenlinux.com/2009/11/12/an-example-of-regular-expression
<p>前篇只记录了一些正则表达式,没有例子来说明。今天说个简单的例子。</p>
<p>squid中定义refresh_pattern,客户要求有http://www.a.com/b/0/到http://www.a.com/b/20/一共21个目录下的所有文件缓存一定时间。起先随意写了http://www.a.com/b/.*,结果在access.log里发现http://www.a.com/b/下,还有很多除了0-20以外的目录。这就没办法了,只能改。</p>
<p>如果一路写上二十一条refresh,实在麻烦。于是改琢磨正则匹配。</p>
<table>
<tbody>
<tr>
<td>最后结果是http://www.a.com/b/(1{0,1}[0-9]</td>
<td>20)/.*</td>
</tr>
</tbody>
</table>
<div class="highlighter-rouge"><pre class="highlight"><code>其中1{0,1}是一部分,{}规定之前的“1”占用0-1位;
[0-9]是第二部分,表示这一位上为0-9的任意一个数字;
两个合在一起,就是(0位)1[0-9]=0-9,(1位)1[0-9]=10-19,也就是0-19;
最后第三部分,单独的20;
全部就是0-20了。
</code></pre>
</div>
shell技巧——getopts
2009-11-04T00:00:00+08:00
bash
http://chenlinux.com/2009/11/04/intro-getopts
<p>在写sh脚本的时候,常常需要运行时输入一些数据。之前已经知道用基本的$*,执行的情况,大概就是$0 $1 $2 $3……<br />
那么,那些系统命令里的参数又是怎么做出来的呢?我们自己的脚本如何搞出来$0-$1的效果呢?这就是getopts的作用了。举例如下:<br />
<code class="highlighter-rouge">bash
#!/bin/bash
echo "OPTIND starts at $OPTIND"
while getopts ":pq:" optname
do
case "$optname" in
"p")
echo "Option $optname is specified"
;;
"q")
echo "Option $optname has value $OPTARG"
;;
"?")
echo "Unknown option $OPTARG"
;;
":")
echo "No argument value for option $OPTARG"
;;
*)
# Should not occur
echo "Unknown error while processing options"
;;
esac
echo "OPTIND is now $OPTIND"
done
</code><br />
在使用getopts命令的时候,shell会自动产生两个变量OPTIND和OPTARG。</p>
<p>OPTIND初始值为1,其含义是下一个待处理的参数的索引。只要存在,getopts命令返回true,所以一般getopts命令使用while循环;</p>
<p>OPTARG是当getopts获取到其期望的参数后存入的位置。而如果不在其期望内,则$optname被设为?并将该意外值存入OPTARG;如果$optname需要拥有具体设置值而实际却没有,则$optname被设为:并将丢失设置值的optname存入OPTARG;</p>
<p>对于$optname,可以用后标:来表示是否需要值;而前标:则表示是否开启静默模式。</p>
分布式shell程序
2009-11-04T00:00:00+08:00
linux
http://chenlinux.com/2009/11/04/dist-shell
<p>除了用expect+for循环以外,今天偶然看到分布式shell这个概念。随手百度一些资料,放到这里,等过段时间试试~~</p>
<p><a href="http://www.netfort.gr.jp/~dancer/software/dsh.html.en">DSH——dancer’s shell / distributed shell</a></p>
<p>开发者是在debian/ubuntu上做的,依赖libdshconfig,如果要部署在其他linux发行版上,还得好好编译一番。<br />
<a href="http://www.theether.org/pssh/">pssh</a> ——这个系列很全,ssh/scp/rsync/nuke/slurp都有。<br />
这个默认下载是rpm包。<br />
<a href="http://sourceforge.net/apps/mediawiki/clusterssh/index.php?title=Main_Page">cssh</a><br />
这个是用perl编写的,感觉普通使用的时候就像是SecureCRT、XShell这些windows平台上的ssh工具一样标签组管理登陆,但多了一个向组服务器同时发送命令的功能。<br />
还有更多工具,见下:<br />
<a href="http://www.gentoo.org/news/zh_cn/gmn/20080930-newsletter.xml">http://www.gentoo.org/news/zh_cn/gmn/20080930-newsletter.xml</a></p>
awk变量(再续)
2009-11-04T00:00:00+08:00
bash
awk
http://chenlinux.com/2009/11/04/awk-variable-2
<p>在squid自动配置脚本里,用到了sed的/r把一个文件的内容插入另一个文件。今天看到awk对两个文件的处理方法,要通过不少运算,不怎么方便。不过作为加深对NR和FNR的不同的理解,还是有些作用。<br />
先说下NR和FNR的不同。<br />
在一次awk中,NR是从头计算到尾的,而FNR是每打开一个文件,就重新计算:<br />
<code class="highlighter-rouge">bash
[root@raocl ~]# awk '{print NR,FNR,$0}' ts st
1 1 123 456
2 2 abc def
3 3 ABC DEF
4 4 654 321
5 1 123 456
6 2 abc def
7 3 ABC DEF
8 4 654 321
</code><br />
下面转载一个例子:<br />
<code class="highlighter-rouge">bash
[root@raocl ~]# cat a
1000 北京市 地级 北京市 北京市
1100 天津市 地级 天津市 天津市
1210 石家庄市 地级 石家庄市 河北省
1210 晋州市 县级 石家庄市 河北省
1243 滦县 县级 唐山市 河北省
1244 滦南县 县级 唐山市 河北省
[root@raocl ~]# cat b
110000,北京市
120000,天津市
130000,河北省
130131,平山县
130132,元氏县
[root@raocl ~]# awk 'BEGIN{FS="[|,]";OFS=","}NRFNR{print
$1,$2,a[$2]}' a b
110000,北京市,1000
120000,天津市,1100
130000,河北省,
130131,平山县,
130132,元氏县,
</code></p>
<p>解释:<br />
NRFNR也就是到文件b的时候,打印文件b的第1、2列和之前创建的数组a[北京市]等。</p>
awk变量(续)
2009-11-04T00:00:00+08:00
bash
awk
http://chenlinux.com/2009/11/04/awk-variable-1
<p>上回用的是-F(其实如果标准化一点,在BEGIN{}里还可以区分成输入输出的FS和OFS)、NR(当前行数)、NF(当前域数)和$0(当前行全部内容),如果是一般的处理,这些差不多也就够了。</p>
<p>今天再学两个东东,RS和RT。</p>
<p>RS,也就是行分割符;RT,咋翻译,我看了好一会man文档也没搞懂,大概的说,如果RS是单字符的话,RT==RS,如果RS用了正则表达式的话,RT就是当前行RS的内容——也不知道这么说是否准确,目前就理解到这步。注意:RT是GNU awk的扩展功能,所以可能有些平台上不支持。呵呵~~</p>
<p>说实话,理解这个RS颇是花了我不少脑细胞去想象。直到看到一个网页,大概意思是这样:<br />
假如test内容是:<br />
123 456<br />
abc def<br />
ABC DEF<br />
654 321<br />
那么对于类UNIX系统来说,test文件内容其实是123 456\nabc def\nABC DEF\n654 321<br />
awk默认的RS,就是”\n”(默认FS是” “和”\t”即tab),每碰见一个RS,awk就停下来,输入space处理。假如一直没有RS,就输完全部文件为止。<br />
如果在BEGIN{}里另外定义RS的话,要注意的是,这个时候”\n”还不会成为字符出现,而是自动转为默认的FS。</p>
<p>对test的实验过程如下:<br />
<code class="highlighter-rouge">bash
[root@raocl ~]# cat test
123 456
abc def
ABC DEF
654 321
[root@raocl ~]# awk '{print $1}' test
123
abc
ABC
654[root@raocl ~]# awk 'BEGIN{RS="ABC"}{print $1}' test
123
DEF
[root@raocl ~]# awk 'BEGIN{RS="ABC";FS=" "}{print $1}' test
123
DEF
[root@raocl ~]# awk 'BEGIN{RS="ABC";FS="\n"}{print $1}' test
123 456
DEF
[root@raocl ~]# awk 'BEGIN{RS="ABC";FS="abc"}{print $1}' test
123 456
DEF
654 321
[root@raocl ~]# awk 'BEGIN{RS="ABC";FS="\t"}{print $1}' test
123 456
abc def
DEF
654 321
</code><br />
我是似乎明白了,不知道路过我博客的同仁们明白了么?</p>
<p>然后说RT,还是用实验来证明吧:<br />
<code class="highlighter-rouge">bash
[root@raocl ~]# awk 'BEGIN{RS="ABC";FS="\t"}{print $1,RT}' ts
123 456
abc def
ABC
DEF
654 321
[root@raocl ~]# awk 'BEGIN{RS="ABC";FS="\t"}{print $1,RS}' ts
123 456
abc def
ABC
DEF
654 321
ABC
</code><br />
对了,还记得上回取上一行用的办法么?我再试试:<br />
<code class="highlighter-rouge">bash
[root@raocl ~]# awk 'BEGIN{RS="ABC";FS="\t"}{print $1,x}{x=RT}' ts
123 456
abc def
DEF
654 321
ABC
</code><br />
也是打印出来上一行的RT了。<br />
这个都是同一的字符做RS,下面转载一个复杂的正则匹配RS的例子:<br />
<code class="highlighter-rouge">bash
[root@mip blog]# cat TR_file
Sun Jan 2 07:42:56 2000
Database mounted in Exclusive Mode
Completed: ALTER DATABASE MOUNT
Sun Jan 2 07:42:56 2000
Database tested in Exclusive Mode
Completed: ALTER DATABASE MOUNT
abc Jan 2 12:42:56 2000
Database mounted in Exclusive Mode
Completed: ALTER DATABASE MOUNT
Sun Jan 2 23:00:00 2009
Database mounted in Exclusive Mode
Completed: ALTER DATABASE MOUNT
[root@mip blog]# awk -v RS='[[:alpha:]]+ [[:alpha:]]+ [0-9][0-9][0-9]:[0-9][0-9]:[0-9][0-9]' '$0~/mounted/{print s}{s=RT}'
RT_file
Sun Jan 2 07:42:56
abc Jan 2 12:42:56
Sun Jan 2 23:00:00
</code></p>
awk内置函数
2009-11-04T00:00:00+08:00
bash
awk
http://chenlinux.com/2009/11/04/awk-built-in-function
<p>前几篇说awk变量,今天说函数。</p>
<p>看到内置变量的中文翻译如下:</p>
<p>ARGC命令行参数个数 AGRV命令行参数排列 ENVIRON支持队列中系统环境变量的使用 FILENAME浏览文件名<br />
FNR浏览文件的记录数 FS输入域分隔符 NF浏览记录的域个数 NR已读的记录数 OFS输出域分隔符 ORS输出记录分隔符<br />
RS控制记录分隔符</p>
<p>index(s,t) 返回s中字符串t的第一位置<br />
[root@raocl ~]# awk ‘BEGIN {print index(“Sunny”,”ny”)}’<br />
4</p>
<p>length(s) 返回s的长度<br />
[root@raocl ~]# awk ‘BEGIN {print length(“Sunny”)}’<br />
5</p>
<p>match(s,r) 测试s是否包含匹配r的字符串,默认带两个变量RSTART、RLENGTH,分别是开始位置和占用长度<br />
[root@raocl ~]# echo 12|awk ‘$1=”J.Lulu”{print match($1,”u”),RSTART,RLENGTH}’<br />
4 4 1</p>
<p>split(s,a,fs) 以fs为分隔符将s分割输入数组a<br />
[root@raocl ~]# awk ‘BEGIN {print split(“12#345#6789”,myarray,”#”),myarray[2]}’<br />
3 345</p>
<p>substr(s,p) 返回字符串s中从p开始的后缀部分</p>
<p>substr(s,p,n) 返回字符串s中从p开始长度为n的后缀部分<br />
[root@raocl ~]# echo abcdefg|awk ‘{print substr($0,1,length($0)-4)}’<br />
abc</p>
<p>gsub(r,s,t) 在t中用s替代r(不写t就是$0)<br />
(附:sub()函数只替换第一次出现的位置;另,sub/gsub修改字符串,而substr是生成子串,不修改原串)<br />
[root@raocl ~]# echo abc|awk ‘gsub(/ab/,”12”,$0)’<br />
12c</p>
让进程在后台可靠运行的几种方法(转)
2009-11-03T00:00:00+08:00
bash
http://chenlinux.com/2009/11/03/zz-three-ways-to-keep-process-running-in-the-background
<ul>
<li>nohup<br />
nohup 无疑是我们首先想到的办法。顾名思义,nohup 的用途就是让提交的命令忽略 hangup 信号。让我们先来看一下nohup 的帮助信息:<br />
NOHUP(1)<br />
User<br />
Commands<br />
NOHUP(1)<br />
NAME<br />
nohup - run a command immune to hangups, with output to a<br />
non-tty<br />
SYNOPSIS<br />
nohup COMMAND [ARG]…<br />
nohup OPTION<br />
DESCRIPTION<br />
Run COMMAND, ignoring hangup signals.<br />
–help display this help and exit<br />
–version<br />
output version information and exit<br />
可见,nohup 的使用是十分方便的,只需在要处理的命令前加上 nohup 即可,标准输出和标准错误缺省会被重定向到nohup.out文件中。一般我们可在结尾加上”&”来将命令同时放入后台运行,也可用”> filename 2>&1”来更改缺省的重定向文件名。</li>
</ul>
<p>** nohup 示例</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">[</span>root@pvcent107 ~]# nohup ping www.ibm.com &
<span class="o">[</span>1] 3059 nohup: appending output to <span class="sb">`</span>nohup.out<span class="s1">'
[root@pvcent107 ~]# ps -ef |grep 3059
root 3059 984 0 21:06 pts/3 00:00:00 ping www.ibm.com
root 3067 984 0 21:06 pts/3 00:00:00 grep 3059
[root@pvcent107 ~]#
</span></code></pre>
</div>
<p>** hangup 名称的来由</p>
<p>在 Unix 的早期版本中,每个终端都会通过 modem 和系统通讯。当用户 logout 时,modem 就会挂断(hangup)电话。 同理,当 modem 断开连接时,就会给终端发送 hangup 信号来通知其关闭所有子进程。</p>
<ul>
<li>setsid</li>
</ul>
<p>nohup 无疑能通过忽略 HUP 信号来使我们的进程避免中途被中断,但如果我们换个角度思考,如果我们的进程不属于接受HUP信号的终端的子进程,那么自然也就不会受到 HUP 信号的影响了。setsid 就能帮助我们做到这一点。让我们先来看一下 setsid的帮助信息:<br />
SETSID(8)<br />
Linux Programmer’s<br />
Manual<br />
SETSID(8)<br />
NAME<br />
setsid - run a program in a new session<br />
SYNOPSIS<br />
setsid program [ arg … ]<br />
DESCRIPTION<br />
setsid runs a program in a new session.<br />
可见 setsid 的使用也是非常方便的,也只需在要处理的命令前加上 setsid 即可。</p>
<p>** setsid 示例</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">[</span>root@pvcent107 ~]# nohup ping www.ibm.com &
<span class="o">[</span>root@pvcent107 ~]# setsid ping www.ibm.com
<span class="o">[</span>root@pvcent107 ~]# ps -ef |grep www.ibm.com
root 31094 1 0 07:28 ? 00:00:00 ping www.ibm.com
root 31102 29217 0 07:29 pts/4 00:00:00 grep www.ibm.com
<span class="o">[</span>root@pvcent107 ~]#
</code></pre>
</div>
<p>值得注意的是,上例中我们的进程 ID(PID)为31094,而它的父 ID(PPID)为1(即为 init 进程ID),并不是当前终端的进程 ID。请将此例与nohup 例中的父 ID 做比较。</p>
<ul>
<li>&</li>
</ul>
<p>这里还有一个关于 subshell 的小技巧。我们知道,将一个或多个命名包含在“()”中就能让这些命令在子 shell中运行中,从而扩展出很多有趣的功能,我们现在要讨论的就是其中之一。<br />
当我们将”&”也放入“()”内之后,我们就会发现所提交的作业并不在作业列表中,也就是说,是无法通过jobs来查看的。让我们来看看为什么这样就能躲过HUP 信号的影响吧。</p>
<p>** subshell示例</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">[</span>root@pvcent107 ~]# nohup ping www.ibm.com &
<span class="o">[</span>root@pvcent107 ~]# <span class="o">(</span>ping www.ibm.com &<span class="o">)</span>
<span class="o">[</span>root@pvcent107 ~]# ps -ef |grep www.ibm.com
root 16270 1 0 14:13 pts/4 00:00:00 ping www.ibm.com
root 16278 15362 0 14:13 pts/4 00:00:00 grep www.ibm.com
<span class="o">[</span>root@pvcent107 ~]#
</code></pre>
</div>
<p>从上例中可以看出,新提交的进程的父 ID(PPID)为1(init 进程的 PID),并不是当前终端的进程ID。因此并不属于当前终端的子进程,从而也就不会受到当前终端的 HUP 信号的影响了。</p>
<ul>
<li>disown</li>
</ul>
<p>场景: <br />
我们已经知道,如果事先在命令前加上 nohup 或者 setsid 就可以避免 HUP 信号的影响。但是如果我们未加任何处理就已经提交了命令,该如何补救才能让它避免 HUP 信号的影响呢?</p>
<p>解决方法: <br />
这时想加 nohup 或者 setsid 已经为时已晚,只能通过作业调度和 disown 来解决这个问题了。让我们来看一下disown 的帮助信息:<br />
disown [-ar] [-h] [jobspec …]<br />
Without options, each jobspec is<br />
removed from<br />
the table<br />
of<br />
active<br />
jobs.<br />
If the -h option is given, each jobspec is not<br />
removed from the table, but is marked so that SIGHUP is not<br />
sent to the job if the shell receives a SIGHUP. If no jobspec<br />
is present, and neither the -a nor the -r option is supplied,<br />
the current job is used. If no jobspec is supplied, the<br />
-a option means to remove or mark all jobs; the -r option without<br />
a jobspec argument restricts operation to running jobs. The<br />
return value is 0 unless a jobspec does not specify a valid job.</p>
<p>可以看出,我们可以用如下方式来达成我们的目的。</p>
<p>** 用disown -h jobspec 来使某个作业忽略HUP信号。<br />
** 用disown -ah 来使所有的作业都忽略HUP信号。<br />
** 用disown -rh 来使正在运行的作业忽略HUP信号。</p>
<p>需要注意的是,当使用过 disown 之后,会将把目标作业从作业列表中移除,我们将不能再使用jobs来查看它,但是依然能够用ps -ef查找到它。</p>
<p>但是还有一个问题,这种方法的操作对象是作业,如果我们在运行命令时在结尾加了”&”来使它成为一个作业并在后台运行,那么就万事大吉了,我们可以通过jobs命令来得到所有作业的列表。但是如果并没有把当前命令作为作业来运行,如何才能得到它的作业号呢?答案就是用CTRL-z(按住Ctrl键的同时按住z键)了!</p>
<p>CTRL-z 的用途就是将当前进程挂起(Suspend),然后我们就可以用jobs命令来查询它的作业号,再用bg jobspec来将它放入后台并继续运行。需要注意的是,如果挂起会影响当前进程的运行结果,请慎用此方法。</p>
<p>** disown示例1</p>
<p>(如果提交命令时已经用“&amp;”将命令放入后台运行,则可以直接使用“disown”)<br />
<code class="highlighter-rouge">bash
[root@pvcent107 build]# cp -r testLargeFile largeFile &
[1] 4825
[root@pvcent107 build]# jobs
[1]+
Running cp -i -r testLargeFile largeFile &
[root@pvcent107 build]# disown -h %1
[root@pvcent107 build]# ps -ef |grep largeFile
root 4825 968 1 09:46 pts/4 00:00:00 cp -i -r testLargeFile largeFile
root 4853 968 0 09:46 pts/4 00:00:00 grep largeFile
[root@pvcent107 build]#
</code></p>
<p>** disown 示例2</p>
<p>(如果提交命令时未使用“&amp;”将命令放入后台运行,可使用 CTRL-z和“bg”将其放入后台,再使用“disown”)<br />
<code class="highlighter-rouge">bash
[root@pvcent107 build]# cp -r testLargeFile largeFile2
[1]+ Stopped
cp -i -r testLargeFile largeFile2
[root@pvcent107 build]# bg %1
[1]+ cp -i -r testLargeFile largeFile2 &
[root@pvcent107 build]# jobs
[1]+ Running
cp -i -r testLargeFile largeFile2 &
[root@pvcent107 build]# disown -h %1
[root@pvcent107 build]# ps -ef |grep largeFile2
root 5790 5577 1 10:04 pts/3 00:00:00 cp -i -r testLargeFile largeFile2
root 5824 5577 0 10:05 pts/3 00:00:00 grep largeFile2
[root@pvcent107 build]#
</code></p>
<p>灵活运用 CTRL-z</p>
<p>在我们的日常工作中,我们可以用 CTRL-z 来将当前进程挂起到后台暂停运行,执行一些别的操作,然后再用 fg来将挂起的进程重新放回前台(也可用 bg来将挂起的进程放在后台)继续运行。这样我们就可以在一个终端内灵活切换运行多个任务,这一点在调试代码时尤为有用。因为将代码编辑器挂起到后台再重新放回时,光标定位仍然停留在上次挂起时的位置,避免了重新定位的麻烦。</p>
<ul>
<li>screen</li>
</ul>
<p>场景: <br />
我们已经知道了如何让进程免受 HUP 信号的影响,但是如果有大量这种命令需要在稳定的后台里运行,如何避免对每条命令都做这样的操作呢?<br />
解决方法: <br />
此时最方便的方法就是 screen 了。简单的说,screen 提供了 ANSI/VT100的终端模拟器,使它能够在一个真实终端下运行多个全屏的伪终端。screen的参数很多,具有很强大的功能,我们在此仅介绍其常用功能以及简要分析一下为什么使用 screen 能够避免 HUP信号的影响。我们先看一下 screen 的帮助信息:<br />
SCREEN(1)<br />
SCREEN(1)<br />
NAME<br />
screen - screen manager with VT100/ANSI terminal emulation<br />
SYNOPSIS<br />
screen [ -options ] [ cmd [ args ] ]<br />
screen -r [[pid.]tty[.host]]<br />
screen -r sessionowner/[[pid.]tty[.host]]<br />
DESCRIPTION<br />
Screen is a full-screen window manager that multiplexes a physical<br />
terminal between several processes (typically interactive shells).<br />
Each virtual terminal provides the functions of a DEC VT100 terminal<br />
and, in addition, several control functions from the ISO 6429 (ECMA 48,<br />
ANSI X3.64) and ISO 2022 standards (e.g. insert/delete line and<br />
support for multiple character sets).<br />
There is a scrollback history buffer for each<br />
virtual terminal and a copy-and-paste mechanism that<br />
allows moving text regions between windows.</p>
<p>使用 screen 很方便,有以下几个常用选项:</p>
<p>** 用screen -dmS session name 来建立一个处于断开模式下的会话(并指定其会话名)。<br />
** 用screen -list 来列出所有会话。<br />
** 用screen -r session name 来重新连接指定会话。<br />
** 用快捷键CTRL-a d 来暂时断开当前会话。</p>
<p>** screen 示例<br />
<code class="highlighter-rouge">bash
[root@pvcent107 ~]# screen -dmS Urumchi
[root@pvcent107 ~]# screen -list
There is a screen on: 12842.Urumchi (Detached)
1 Socket in /tmp/screens/S-root.
[root@pvcent107 ~]# screen -r
Urumchi
</code><br />
当我们用“-r”连接到 screen 会话后,我们就可以在这个伪终端里面为所欲为,再也不用担心 HUP 信号会对我们的进程造成影响,也不用给每个命令前都加上“nohup”或者“setsid”了。这是为什么呢?让我来看一下下面两个例子吧。<br />
<em>** 未使用 screen 时新进程的进程树<br />
<code class="highlighter-rouge">bash
[root@pvcent107 ~]# ping www.google.com &amp;amp;
[1] 9499
[root@pvcent107 ~]# pstree -H 9499
init─┬─Xvnc
├─acpid
├─atd
├─2*[sendmail]
├─sshd─┬─sshd───bash───pstree
│
└─sshd───bash───ping
</code><br />
我们可以看出,未使用 screen 时我们所处的 bash 是 sshd 的子进程,当 ssh 断开连接时,HUP信号自然会影响到它下面的所有子进程(包括我们新建立的 ping 进程)。<br />
**</em> 使用了 screen 后新进程的进程树<br />
<code class="highlighter-rouge">bash
[root@pvcent107 ~]# screen -r Urumchi
[root@pvcent107 ~]# ping www.ibm.com &amp;amp;
[1] 9488
[root@pvcent107 ~]# pstree -H 9488
init─┬─Xvnc
├─acpid
├─atd
├─screen───bash───ping
├─2*[sendmail]
</code><br />
而使用了 screen 后就不同了,此时 bash 是 screen 的子进程,而 screen 是init(PID为1)的子进程。那么当 ssh 断开连接时,HUP 信号自然不会影响到 screen 下面的子进程了。</p>
<ul>
<li>总结</li>
</ul>
<p>现在几种方法已经介绍完毕,我们可以根据不同的场景来选择不同的方案。nohup/setsid 无疑是临时需要时最方便的方法,disown能帮助我们来事后补救当前已经在运行了的作业,而 screen 则是在大批量操作时不二的选择了。<br />
* 参考资料<br />
** “系统管理员工具包:进程管理技巧”(developerWorks 中国,2006 年 5 月)介绍了 Linux进程管理的更多技巧。<br />
** “Linux 技巧:使用 screen 管理你的远程会话”(developerWorks 中国,2007 年 7 月)介绍了screen 的更多技巧。<br />
* 在 developerWorks 中国网站 Linux 专区中学习更多 Linux 方面的知识。<br />
* 关于作者<br />
申毅,IBM 中国软件开发中心 WebSphere Portal 部门软件工程师。<br />
<a href="http://www.ibm.com/developerworks/cn/linux/l-cn-nohup/index.html">原文地址</a></p>
Linux命令行系统性能检测工具(转)
2009-11-03T00:00:00+08:00
linux
http://chenlinux.com/2009/11/03/zz-linux-performance-command-tools
<p>※注:下面附图的命令输出信息,以红旗DC Server 5.0 for x86 Sp1为基础平台,可能在不同的操作系统或核心版本有较大区别,对比时请留意。</p>
<ul>
<li>uptime</li>
</ul>
<p>Uptime 命令的显示结果包括服务器已经运行了多长时间,有多少登陆用户和对服务器性能的总体评估(load average)。load average值分别记录了上个1分钟,5分钟和15分钟间隔的负载情况,load average不是一个百分比,而是在队列中等待执行的进程的数量。如果进程要求CPU时间被阻塞(意味着CPU没有时间处理它),load average值将增加。另一方面,如果每个进程都可以立刻得到访问CPU的时间,这个值将减少。<br />
kernel下的load average的最佳值是1,这说明每个进程都可以立刻被CPU处理,当然,更低不会有问题,只说明浪费了一部分的资源。但在不同的系统间这个值也是不同的,例如一个单CPU的工作站,load average为1或者2都是可以接受的,而在一个多CPU的系统中这个值应除以物理CPU的个数,假设CPU个数为4,而load average为8或者10,那结果也是在2多点而已。<br />
<a href="http://www.linuxfly.org/attachment/1169008391_0.jpg"><img title="点击在新窗口中浏览此图片" src="http://www.linuxfly.org/attachment/1169008391_0.jpg" border="0" alt="点击在新窗口中浏览此图片" width="500" /></a><br />
你可以使用uptime判断一个性能问题是出现在服务器上还是网络上。例如,如果一个网络应用运行性能不理想,运行uptime检查系统负载是否比较高,如果不是这个问题更可能出现在你的网络上。</p>
<ul>
<li>top</li>
</ul>
<p>Top命令显示了实际CPU使用情况,默认情况下,它显示了服务器上占用CPU的任务信息并且每5秒钟刷新一次。你可以通过多种方式分类它们,包括PID、时间和内存使用情况。<br />
<a href="http://www.linuxfly.org/attachment/1169008444_0.jpg"><img title="点击在新窗口中浏览此图片" src="http://www.linuxfly.org/attachment/1169008444_0.jpg" border="0" alt="点击在新窗口中浏览此图片" width="500" /></a><br />
下面是输出值的介绍:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>PID:进程标识
USER;进程所有者的用户名
PRI:进程的优先级
NI:nice级别
SIZE:进程占用的内存数量(代码+数据+堆栈)
RSS;进程使用的物理内存数量
SHARE;该进程和其他进程共享内存的数量
STAT:进程的状态:S=休眠状态,R=运行状态,T=停止状态,D=中断休眠状态,Z=僵尸状态
%CPU:共享的CPU使用
%MEM;共享的物理内存
TIME:进程占用CPU的时间
COMMAND:启动任务的命令行(包括参数)
</code></pre>
</div>
<p>** 进程的优先级和nice级别<br />
进程优先级是一个决定进程被CPU执行优先顺序的参数,内核会根据需要调整这个值。Nice值是一个对优先权的限制。进程优先级的值不能低于nice值。(nice值越低优先级越高)<br />
进程优先级是无法去手动改变的,只有通过改变nice值去间接的调整进程优先级。如果一个进程运行的太慢了,你可以通过指定一个较低的nice值去为它分配更多的CPU资源。当然,这意味着其他的一些进程将被分配更少的CPU资源,运行更慢一些。Linux支持nice值的范围是19(低优先级)到-20(高优先级),默认的值是0。如果需要改变一个进程的nice值为负数(高优先级),必须使用su命令登陆到root用户。下面是一些调整nice值的命令示例,</p>
<p>以nice值-5开始程序xyz</p>
<p>#nice –n -5 xyz</p>
<p>改变已经运行的程序的nice值</p>
<p>#renice level pid</p>
<p>将pid为2500的进程的nice值改为10</p>
<p>#renice 10 2500</p>
<p>** 僵尸进程<br />
当一个进程被结束,在它结束之前通常需要用一些时间去完成所有的任务(比如关闭打开的文件),在一个很短的时间里,这个进程的状态为僵尸状态。在进程完成所有关闭任务之后,会向父进程提交它关闭的信息。有些情况下,一个僵尸进程不能关闭它自己,这时这个进程状态就为z(zombie)。不能使用kill命令杀死僵尸进程,因为它已经标志为“dead”。如果你无法摆脱一个僵尸进程,你可以杀死它的父进程,这个僵尸进程也就消失了。然而,如果父进程是init进程,你不能杀死init进程,因为init是一个重要的系统进程,这种情况下你只能通过一次重新启动服务器来摆脱僵尸进程。也必须分析应用为什么会导致僵死?</p>
<ul>
<li>iostat</li>
</ul>
<p>iostat是sysstat包的一部分。Iostat显示自系统启动后的平均CPU时间(与uptime类似),它也可以显示磁盘子系统的使用情况,iostat可以用来监测CPU利用率和磁盘利用率。<br />
<a href="http://www.linuxfly.org/attachment/1169008526_0.jpg"><img title="点击在新窗口中浏览此图片" src="http://www.linuxfly.org/attachment/1169008526_0.jpg" border="0" alt="点击在新窗口中浏览此图片" width="500" /></a></p>
<p>CPU利用率分四个部分:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>%user:user level(应用)的CPU占用率情况
%nice:加入nice优先级的user level的CPU占用率情况
%sys:system level(内核)的CPU占用情况
%idle:空闲的CPU资源情况
</code></pre>
</div>
<p>磁盘占用率有下面几个部分:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Device:块设备名
Tps:设备每秒进行传输的数量(每秒的I/O请求)。多个单独的I/O请求可以被组成一个传输操作,因为一个传输操作可以是不同的容量。
Blk_read/s, Blk_wrtn/s:该设备每秒读写的块的数量。块可能为不同的容量。
Blk_read, Blk_wrtn:自系统启动以来读写的块设备的总量。
</code></pre>
</div>
<p>块的大小<br />
块可能为不同的容量。块的大小一般为1024、2048、4048byte。可通过tune2fs或dumpe2fs获得:<br />
<code class="highlighter-rouge">bash
[root@rfgz ~]# tune2fs -l /dev/hda1|grep 'Block size'
Block size: 4096
[root@rfgz ~]# dumpe2fs -h /dev/hda1|grep 'Block size'
dumpe2fs 1.35 (28-Feb-2004)
Block size: 4096
</code></p>
<ul>
<li>Vmstat<br />
Vmstat命令提供了对进程、内存、页面I/O块和CPU等信息的监控,vmstat可以显示检测结果的平均值或者取样值,取样模式可以提供一个取样时间段内不同频率的监测结果。<br />
<a href="http://www.linuxfly.org/attachment/1169008594_0.jpg"><img title="点击在新窗口中浏览此图片" src="http://www.linuxfly.org/attachment/1169008594_0.jpg" border="0" alt="点击在新窗口中浏览此图片" width="500" /></a></li>
</ul>
<p>注:在取样模式中需要考虑在数据收集中可能出现的误差,将取样频率设为比较低的值可以尽可能的减小误差的影响。<br />
下面介绍一下各列的含义</p>
<div class="highlighter-rouge"><pre class="highlight"><code>·process(procs)
r:等待运行时间的进程数量
b:处在不可中断睡眠状态的进程
w:被交换出去但是仍然可以运行的进程,这个值是计算出来的
·memoryswpd:虚拟内存的数量
free:空闲内存的数量
buff:用做缓冲区的内存数量
·swap
si:从硬盘交换来的数量
so:交换到硬盘去的数量
·IO
bi:向一个块设备输出的块数量
bo:从一个块设备接受的块数量
·system
in:每秒发生的中断数量, 包括时钟
cs:每秒发生的context switches的数量
·cpu(整个cpu运行时间的百分比)
us:非内核代码运行的时间(用户时间,包括nice时间)
sy:内核代码运行的时间(系统时间)
id:空闲时间,在Linux 2.5.41之前的内核版本中,这个值包括I/O等待时间;
wa:等待I/O操作的时间,在Linux 2.5.41之前的内核版本中这个值为0
</code></pre>
</div>
<p>Vmstat命令提供了大量的附加参数,下面列举几个十分有用的参数:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>·m:显示内核的内存利用率
·a:显示内存页面信息,包括活跃和不活跃的内存页面
·n:显示报头行,这个参数在使用取样模式并将命令结果输出到一个文件时非常有用。例如root#vmstat –n 2 10以2秒的频率显示10输出结果
·当使用-p {分区}时,vmstat提供对I/O结果的统计
</code></pre>
</div>
<p><a href="http://www.linuxfly.org/attachment/1169008668_0.jpg"><img title="点击在新窗口中浏览此图片" src="http://www.linuxfly.org/attachment/1169008668_0.jpg" border="0" alt="点击在新窗口中浏览此图片" width="500" /></a><br />
<a href="http://www.linuxfly.org/attachment/1169008747_0.jpg"><img title="点击在新窗口中浏览此图片" src="http://www.linuxfly.org/attachment/1169008747_0.jpg" border="0" alt="点击在新窗口中浏览此图片" width="500" /></a></p>
<ul>
<li>ps和pstree</li>
</ul>
<p>ps 和pstree命令是系统分析最常用的基本命令,ps命令提供了一个正在运行的进程的列表,列出进程的数量取决于命令所附加的参数。例如ps –A 命令列出所有进程和它们相应的进程ID(PID),进程的PID是使用其他一些工具之前所必须了解的,例如pmap或者renice。<br />
在运行java应用的系统上,ps –A 命令的输出很容易就会超过屏幕的显示范围,这样就很难得到所有进程的完整信息。这时,使用pstree命令可以以树状结构来显示所有的进程信息并且可以整合子进程的信息。Pstree命令对分析进程的来源十分有用。<br />
<a href="http://www.linuxfly.org/attachment/1169008793_0.jpg"><img title="点击在新窗口中浏览此图片" src="http://www.linuxfly.org/attachment/1169008793_0.jpg" border="0" alt="点击在新窗口中浏览此图片" /></a></p>
<ul>
<li>Numastat</li>
</ul>
<p>随着NUMA架构的不断发展,例如eServer xSeries 445及其后续产品eServer xSeries 460,现在NUMA架构已经成为了企业级数据中心的主流。然而,NUMA架构在性能调优方面面临了新的挑战,例如内存分配的问题在NUMA系统之前并没人感兴趣,而Numastat命令提供了一个监测NUMA架构的工具。Numastat命令提供了本地内存与远程内存使用情况的对比和各个节点的内存使用情况。Numa_miss列显示分配失败的本地内存,numa_foreign列显示分配远程内存(访问速度慢)信息,过多的调用远程内存将增加系统的延迟从而影响整个系统的性能。使运行在一个节点上的进程都访问本地内存将极大的改善系统的性能。<br />
<a href="http://www.linuxfly.org/attachment/1169008814_0.jpg"><img title="点击在新窗口中浏览此图片" src="http://www.linuxfly.org/attachment/1169008814_0.jpg" border="0" alt="点击在新窗口中浏览此图片" width="500" /></a></p>
<ul>
<li>sar</li>
</ul>
<p>sar程序也是sysstat安装包的一部分。sar命令用于收集、报告和保存系统的信息。Sar命令由三个应用组成:sar,用与显示数据;sa1和sa2,用于收集和存储数据。默认情况下,系统会在crontab中加入自动收集和分析的操作:</p>
<div class="highlighter-rouge"><pre class="highlight"><code><span class="o">[</span>root@rfgz ~]# cat /etc/cron.d/sysstat
<span class="c"># run system activity accounting tool every 10 minutes</span>
<span class="k">*</span>/10 <span class="k">*</span> <span class="k">*</span> <span class="k">*</span> <span class="k">*</span> root /usr/lib/sa/sa1 1 1
<span class="c"># generate a daily summary of process accounting at 23:53</span>
53 23 <span class="k">*</span> <span class="k">*</span> <span class="k">*</span> root /usr/lib/sa/sa2 -A
</code></pre>
</div>
<p>sar命令所生成的数据保存在/var/log/sa/目录下,数据按照时间保存,可以根据时间来查询相应的性能数据。<br />
你也可以使用sar在命令行下得到一个实时的执行结果,收集的数据可以包括CPU利用率、内存页面、网络I/O等等。下面的命令表示用sar执行5次,间隔时间为3秒:<br />
<a href="http://www.linuxfly.org/attachment/1169008875_0.jpg"><img title="点击在新窗口中浏览此图片" src="http://www.linuxfly.org/attachment/1169008875_0.jpg" border="0" alt="点击在新窗口中浏览此图片" width="500" /></a><br />
八、free<br />
free命令显示系统的所有内存的使用情况,包括空闲内存、被使用的内存和交换内存空间。Free命令显示也包括一些内核使用的缓存和缓冲区的信息。<br />
当使用free命令的时候,需要记住linux的内存结构和虚拟内存的管理方法,比如空闲内存数量的限制,还有swap空间的使用并不标志一个内存瓶颈的出现。<br />
<a href="http://www.linuxfly.org/attachment/1169008909_0.jpg"><img title="点击在新窗口中浏览此图片" src="http://www.linuxfly.org/attachment/1169008909_0.jpg" border="0" alt="点击在新窗口中浏览此图片" width="500" /></a><br />
Free命令有用的参数:</p>
<p>引用</p>
<p>·-b,-k,-m和-g分别按照bytes, kilobytes, megabytes, gigabytes显示结果。<br />
·-l区别显示low和high内存<br />
·-c {count}显示free输出的次数</p>
<p>九、Pmap<br />
pmap命令显示一个或者多个进程使用内存的数量,你可以用这个工具来确定服务器上哪个进程占用了过多的内存从而导致内存瓶颈。<br />
<a href="http://www.linuxfly.org/attachment/1169008975_0.jpg"><img title="点击在新窗口中浏览此图片" src="http://www.linuxfly.org/attachment/1169008975_0.jpg" border="0" alt="点击在新窗口中浏览此图片" width="500" /></a><br />
十、Strace<br />
strace截取和记录进程的系统调用信息,还包括进程接受的命令信号。这是一个有用的诊断和调试工具,系统管理员可以通过strace来解决程序上的问题。<br />
命令格式,需要指定需要监测的进程ID。这个多为开发人员使用。</p>
<p>strace -p <pid></pid></p>
<p>十一、ulimit<br />
可以通过ulimit来控制系统资源的使用。请看以前的日志:<a href="http://www.linuxfly.org/post/73.htm">使用ulimit和proc去调整系统参数</a><br />
十二、Mpstat<br />
mpstat命令也是sysstat包的一部分。Mpstat命令用于监测一个多CPU系统中每个可用CPU的情况。Mpstat命令可以显示每个CPU或者所有CPU的运行情况,同时也可以像vmstat命令那样使用参数进行一定频率的采样结果的监测。<br />
<a href="http://www.linuxfly.org/attachment/1169009042_0.jpg"><img title="点击在新窗口中浏览此图片" src="http://www.linuxfly.org/attachment/1169009042_0.jpg" border="0" alt="点击在新窗口中浏览此图片" width="500" /></a></p>
<p> </p>
<p> </p>
<p> </p>
fuser命令(转)
2009-11-03T00:00:00+08:00
bash
http://chenlinux.com/2009/11/03/zz-fuser-usage
<p>fuser:使用文件或者套节字来表示识别进程。我常用的他的两个功能:查看我需要的进程和我要杀死我查到的进程。</p>
<p>比如当你想umount光驱的时候,结果系统提示你设备正在使用或者正忙,可是你又找不到到底谁使用了他。这个时候fuser可派上用场了。<br />
<code class="highlighter-rouge">bash
[root@lancy sbin]# umount /media/cdrom
umount: /media/cdrom: device is busy
umount: /media/cdrom: device is busy
eject: unmount of `/media/cdrom' failed
[root@lancy sbin]# fuser /mnt/cdrom
/mnt/cdrom: 4561c 5382c
[root@lancy sbin]# ps -ef |egrep '(4561|5382)' |grep -v grep
root 4561 4227 0 20:13 pts/1 00:00:00 bash
root 5382 4561 0 21:42 pts/1 00:00:00 vim Autorun.inf
</code><br />
示例中,我想弹出光驱,系统告诉我设备忙着,于是采用fuser命令,参数是你文件或scoket,fuser将查出那些使用了他。</p>
<p>4561c,5382c表示目前用两个进程在占用着/mnt/cdrom,分别是4561,5382,进程ID后的字母表示占用资源的方式,有下面几种表示:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>c 当前路径(current directory.)我的理解是表示这个资源的占用是以文件目录方式,也就是进进入了需要释放的资源的路径,这是最常用的资源占用方式。
e 正在运行可执行文件(executable being run.),比如运行了光盘上的某个程序
f 打开文件( open file),缺省模式下f忽略。所以上面的例子中,虽然是开打了光盘上的Autorun.inf文件,但是给出的标识是c,而不是f。
r root目录(root directory).没有明白什么意思,难道是说进入了/root这个特定目录?
m mmap文件或者共享库( mmap’ed file or shared library).这应该是说某个进程使用了你要释放的资源的某个共享文件。
</code></pre>
</div>
<p>在查找的同时,你还可定指定一些参数,比如</p>
<div class="highlighter-rouge"><pre class="highlight"><code>-k 杀死这些正在访问这些文件的进程。除非使用-signal修改信号,否则将发送SIGKILL信号。
-i 交互模式
-l 列出所有已知的信号名称。
-n 空间,选择不同的名字空间,可是file,udp,tcp。默认是file,也就是文件。
-signal 指定发送的信号,而不是缺省的SIGKILL
-4 仅查询IPV4套接字
-6 仅查询IPV6套接字
- 重置所有的选项,将信息设回SIGKILL
</code></pre>
</div>
<p>再看下面的例子<br />
<code class="highlighter-rouge">bash
[root@lancy sbin]# fuser -l
HUP INT QUIT ILL TRAP ABRT IOT BUS FPE KILL USR1 SEGV USR2 PIPE
ALRM TERM
STKFLT CHLD CONT STOP TSTP TTIN TTOU URG XCPU XFSZ VTALRM PROF
WINCH IO PWR SYS
UNUSED
</code><br />
现在我们试试fuser -k的威力:<br />
<code class="highlighter-rouge">bash
[root@lancy sbin]# fuser -k /mnt/cdrom
/mnt/cdrom: 4561c 5382c
kill 5382: 没有那个进程
No automatic removal. Please use umount /media/cdrom
[root@lancy sbin]# eject
</code><br />
套节字方式的使用:<br />
<code class="highlighter-rouge">bash
[root@lancy sbin]# fuser -4 -n tcp 3306
here: 3306
3306/tcp: 5595
[root@lancy sbin]# ps -ef |grep 5595 |grep -v grep
mysql 5595 5563 0 22:24 pts/0 00:00:00 /usr/libexec/mysqld
--defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql
--user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-locking
--socket=/var/lib/mysql/mysql.sock
[root@lancy sbin]# fuser -4 -n tcp 80
here: 80
80/tcp: 5685 5688 5689 5690 5691 5692 5693 5694 5695
</code></p>
dd命令使用详解(转)
2009-11-03T00:00:00+08:00
bash
http://chenlinux.com/2009/11/03/zz-dd-usage
<ol>
<li>命令简介</li>
</ol>
<p>dd 的主要选项:<br />
指定数字的地方若以下列字符结尾乘以相应的数字:<br />
b=512, c=1, k=1024, w=2, xm=number m</p>
<p>if=file<br />
输入文件名,缺省为标准输入。<br />
of=file<br />
输出文件名,缺省为标准输出。<br />
ibs=bytes<br />
一次读入 bytes 个字节(即一个块大小为 bytes 个字节)。<br />
obs=bytes<br />
一次写 bytes 个字节(即一个块大小为 bytes 个字节)。<br />
bs=bytes<br />
同时设置读写块的大小为 bytes ,可代替 ibs 和 obs 。<br />
cbs=bytes<br />
一次转换 bytes 个字节,即转换缓冲区大小。<br />
skip=blocks<br />
从输入文件开头跳过 blocks 个块后再开始复制。<br />
seek=blocks<br />
从输出文件开头跳过 blocks 个块后再开始复制。(通常只有当输出文件是磁盘或磁带时才有效)。<br />
count=blocks<br />
仅拷贝 blocks 个块,块大小等于 ibs 指定的字节数。<br />
conv=conversion[,conversion…]<br />
用指定的参数转换文件。</p>
<p>转换参数:<br />
ascii 转换 EBCDIC 为 ASCII。<br />
ebcdic 转换 ASCII 为 EBCDIC。<br />
ibm 转换 ASCII 为 alternate EBCDIC.<br />
block 把每一行转换为长度为 cbs 的记录,不足部分用空格填充。<br />
unblock 使每一行的长度都为 cbs ,不足部分用空格填充。<br />
lcase 把大写字符转换为小写字符。<br />
ucase 把小写字符转换为大写字符。<br />
swab 交换输入的每对字节。<br />
noerror 出错时不停止。<br />
notrunc 不截短输出文件。<br />
sync 把每个输入块填充到ibs个字节,不足部分用空(NUL)字符补齐。</p>
<p>2.实例分析</p>
<p>2.1.数据备份与恢复</p>
<p>2.1.1整盘数据备份与恢复<br />
备份:<br />
dd if=/dev/hdx of=/dev/hdy<br />
将本地的/dev/hdx整盘备份到/dev/hdy<br />
dd if=/dev/hdx of=/path/to/image<br />
将/dev/hdx全盘数据备份到指定路径的image文件<br />
dd if=/dev/hdx | gzip >/path/to/image.gz<br />
备份/dev/hdx全盘数据,并利用gzip工具进行压缩,保存到指定路径<br />
恢复:<br />
dd if=/path/to/image of=/dev/hdx<br />
将备份文件恢复到指定盘<br />
gzip -dc /path/to/image.gz | dd of=/dev/hdx<br />
将压缩的备份文件恢复到指定盘</p>
<p>2.1.2.利用netcat远程备份<br />
dd if=/dev/hda bs=16065b | netcat 1234<br />
在源主机上执行此命令备份/dev/hda<br />
netcat -l -p 1234 | dd of=/dev/hdc bs=16065b<br />
在目的主机上执行此命令来接收数据并写入/dev/hdc<br />
netcat -l -p 1234 | bzip2 > partition.img<br />
netcat -l -p 1234 | gzip > partition.img<br />
以上两条指令是目的主机指令的变化分别采用bzip2 gzip对数据进行压缩,并将备份文件保存在当前目录。</p>
<p>2.1.3.备份MBR<br />
备份:<br />
dd if=/dev/hdx of=/path/to/image count=1<br />
bs=512<br />
备份磁盘开始的512Byte大小的MBR信息到指定文件<br />
恢复:<br />
dd if=/path/to/image of=/dev/hdx<br />
将备份的MBR信息写到磁盘开始部分</p>
<p>2.1.4.备份软盘<br />
dd if=/dev/fd0 of=disk.img count=1<br />
bs=1440k<br />
将软驱数据备份到当前目录的disk.img文件</p>
<p>2.1.5.拷贝内存资料到硬盘<br />
dd if=/dev/mem of=/root/mem.bin<br />
bs=1024<br />
将内存里的数据拷贝到root目录下的mem.bin文件</p>
<p>2.1.6.从光盘拷贝iso镜像<br />
dd if=/dev/cdrom of=/root/cd.iso<br />
拷贝光盘数据到root文件夹下,并保存为cd.iso文件</p>
<p>2.2.增加Swap分区文件大小<br />
dd if=/dev/zero of=/swapfile bs=1024 count=262144<br />
创建一个足够大的文件(此处为256M)<br />
mkswap /swapfile<br />
把这个文件变成swap文件<br />
swapon /swapfile<br />
启用这个swap文件<br />
/swapfile swap swap defaults 0 0<br />
在每次开机的时候自动加载swap文件, 需要在 /etc/fstab文件中增加一行</p>
<p>2.3.销毁磁盘数据<br />
dd if=/dev/urandom of=/dev/hda1<br />
利用随机的数据填充硬盘,在某些必要的场合可以用来销毁数据。执行此操作以后,/dev/hda1将无法挂载,创建和拷贝操作无法执行。</p>
<p>2.4.磁盘管理</p>
<p>2.4.1.得到最恰当的block size<br />
dd if=/dev/zero bs=1024 count=1000000<br />
of=/root/1Gb.file<br />
dd if=/dev/zero bs=2048 count=500000<br />
of=/root/1Gb.file<br />
dd if=/dev/zero bs=4096 count=250000<br />
of=/root/1Gb.file<br />
dd if=/dev/zero bs=8192 count=125000<br />
of=/root/1Gb.file<br />
通过比较dd指令输出中所显示的命令执行时间,即可确定系统最佳的blocksize大小</p>
<p>2.4.2测试硬盘读写速度<br />
dd if=/root/1Gb.file bs=64k | dd<br />
of=/dev/null<br />
dd if=/dev/zero of=/root/1Gb.file bs=1024<br />
count=1000000<br />
通过上两个命令输出的执行时间,可以计算出测试硬盘的读/写速度</p>
<p>2.4.3.修复硬盘<br />
dd if=/dev/sda of=/dev/sda<br />
当硬盘较长时间(比如1,2年)放置不使用后,磁盘上会产生magnetic fluxpoint。当磁头读到这些区域时会遇到困难,并可能导致I/O错误。当这种情况影响到硬盘的第一个扇区时,可能导致硬盘报废。上边的命令有可能使这些数据起死回生。且这个过程是安全,高效的</p>
curl使用简单介绍(转)
2009-11-03T00:00:00+08:00
bash
curl
http://chenlinux.com/2009/11/03/zz-curl-usage
<p>原文地址: <a href="http://www.linuxidc.com/Linux/2008-01/10891p2.htm">http://www.linuxidc.com/Linux/2008-01/10891p2.htm</a><br />
Curl是Linux下一个很强大的http命令行工具,其功能十分强大。</p>
<ol>
<li>
<p>二话不说,先从这里开始吧!<br />
$ curl http://www.linuxidc.com<br />
回车之后,www.linuxidc.com的html就稀里哗啦地显示在屏幕上了~</p>
</li>
<li>
<p>嗯,要想把读过来页面存下来,是不是要这样呢?<br />
$ curl http://www.linuxidc.com > page.html<br />
当然可以,但不用这么麻烦的!<br />
用curl的内置option就好,存下http的结果,用这个option: -o<br />
$ curl -o page.html http://www.linuxidc.com<br />
这样,你就可以看到屏幕上出现一个下载页面进度指示。等进展到100%,自然就 OK咯</p>
</li>
<li>
<p>什么什么?!访问不到?肯定是你的proxy没有设定了。<br />
使用curl的时候,用这个option可以指定http访问所使用的proxy服务器及其端口: -x<br />
$ curl -x 123.45.67.89:1080 -o page.html http://www.linuxidc.com</p>
</li>
<li>
<p>访问有些网站的时候比较讨厌,他使用cookie来记录session信息。<br />
像IE/NN这样的浏览器,当然可以轻易处理cookie信息,但我们的curl呢?…..<br />
我们来学习这个option: -D<br />
这个是把http的response里面的cookie信息存到一个特别的文件中去<br />
$ curl -x 123.45.67.89:1080 -o page.html -D cookie0001.txt http://www.linuxidc.com<br />
这样,当页面被存到page.html的同时,cookie信息也被存到了cookie0001.txt里面了</p>
</li>
</ol>
<p>5)那么,下一次访问的时候,如何继续使用上次留下的cookie信息呢?要知道,很多网站都是靠监视你的cookie信息,来判断你是不是不按规矩访问他们的网站的。<br />
这次我们使用这个option来把上次的cookie信息追加到http request里面去: -b<br />
$ curl -x 123.45.67.89:1080 -o page1.html -D cookie0002.txt -b cookie0001.txt http://www.linuxidc.com<br />
这样,我们就可以几乎模拟所有的IE操作,去访问网页了!</p>
<p>6)稍微等等<br />
~我好像忘记什么了<br />
~<br />
对了!是浏览器信息<br />
有些讨厌的网站总要我们使用某些特定的浏览器去访问他们,有时候更过分的是,还要使用某些特定的版本<br />
NND,哪里有时间为了它去找这些怪异的浏览器呢!?<br />
好在curl给我们提供了一个有用的option,可以让我们随意指定自己这次访问所宣称的自己的浏览器信息: -A<br />
$ curl -A “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)” -x 123.45.67.89:1080 -o page.html -D cookie0001.txt http://www.linuxidc.com<br />
这样,服务器端接到访问的要求,会认为你是一个运行在Windows 2000上的IE6.0,嘿嘿嘿,其实也许你用的是苹果机呢!<br />
而”Mozilla/4.73 [en] (X11; U; Linux 2.2; 15 i686”则可以告诉对方你是一台PC上跑着的Linux,用的是Netscape 4.73,呵呵呵</p>
<p>7) 另外一个服务器端常用的限制方法,就是检查http访问的referer。比如你先访问首页,再访问里面所指定的下载页,这第二次访问的referer地址就是第一次访问成功后的页面地址。这样,服务器端只要发现对下载页面某次访问的referer地址不是首页的地址,就可以断定那是个盗连了~讨厌讨厌~我就是要盗链~!!<br />
幸好curl给我们提供了设定referer的option: -e<br />
$ curl -A “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)” -x 123.45.67.89:1080 -e “mail.linuxidc.com” -o page.html -D cookie0001.txt http://www.linuxidc.com<br />
这样,就可以骗对方的服务器,你是从mail.linuxidc.com点击某个链接过来的了,呵呵呵</p>
<p>8)写着写着发现漏掉什么重要的东西了!——- 利用curl 下载文件<br />
刚才讲过了,下载页面到一个文件里,可以使用 -o ,下载文件也是一样。比如,<br />
$ curl -o 1.jpg http://cgi2.tky.3web.ne.jp/~zzh/screen1.JPG<br />
这里教大家一个新的option: -O 大写的O,这么用:<br />
$ curl -O http://cgi2.tky.3web.ne.jp/~zzh/screen1.JPG<br />
这样,就可以按照服务器上的文件名,自动存在本地了!<br />
再来一个更好用的。<br />
如果screen1.JPG以外还有screen2.JPG、screen3.JPG、….、screen10.JPG需要下载,难不成还要让我们写一个script来完成这些操作?<br />
不干!<br />
在curl里面,这么写就可以了:<br />
$ curl -O http://cgi2.tky.3web.ne.jp/~zzh/screen[1-10].JPG<br />
呵呵呵,厉害吧?! ~</p>
<p>9)再来,我们继续讲解下载!<br />
$ curl -O <a href="http://cgi2.tky.3web.ne.jp/~{zzh,nick}/[001-201].JPG">http://cgi2.tky.3web.ne.jp/~{zzh,nick}/[001-201].JPG</a><br />
这样产生的下载,就是<br />
~zzh/001.JPG<br />
~zzh/002.JPG<br />
…<br />
~zzh/201.JPG<br />
~nick/001.JPG<br />
~nick/002.JPG<br />
…<br />
~nick/201.JPG<br />
够方便的了吧?哈哈哈<br />
咦?高兴得太早了。<br />
由于zzh/nick下的文件名都是001,002…,201,下载下来的文件重名,后面的把前面的文件都给覆盖掉了 ~<br />
没关系,我们还有更狠的!<br />
$ curl -o #2_#1.jpg http://cgi2.tky.3web.ne.jp/~{zzh,nick}/[001-201].JPG<br />
—这是…..自定义文件名的下载? 对头,呵呵!<br />
这样,自定义出来下载下来的文件名,就变成了这样:原来: ~zzh/001.JPG —-> 下载后:001-zzh.JPG 原来: ~nick/001.JPG —-> 下载后:001-nick.JPG<br />
这样一来,就不怕文件重名啦,呵呵</p>
<p>9)继续讲下载<br />
我们平时在windows平台上,flashget这样的工具可以帮我们分块并行下载,还可以断线续传。curl在这些方面也不输给谁,嘿嘿比如我们下载screen1.JPG中,突然掉线了,我们就可以这样开始续传<br />
$ curl -c -O http://cgi2.tky.3wb.ne.jp/~zzh/screen1.JPG<br />
当然,你不要拿个flashget下载了一半的文件来糊弄我。别的下载软件的半截文件可不一定能用哦 ~</p>
<p>分块下载,我们使用这个option就可以了: -r<br />
举例说明<br />
比如我们有一个http://cgi2.tky.3web.ne.jp/~zzh/zhao1.MP3 要下载(赵老师的电话朗诵 :D)我们就可以用这样的命令:<br />
$ curl -r 0-10240 -o “zhao.part1” http:/cgi2.tky.3web.ne.jp/~zzh/zhao1.MP3 &<br />
$ curl -r 10241-20480 -o “zhao.part1” http:/cgi2.tky.3web.ne.jp/~zzh/zhao1.MP3 &<br />
$ curl -r 20481-40960 -o “zhao.part1” http:/cgi2.tky.3web.ne.jp/~zzh/zhao1.MP3 &<br />
$ curl -r 40961- -o “zhao.part1” http:/cgi2.tky.3web.ne.jp/~zzh/zhao1.MP3<br />
这样就可以分块下载啦。不过你需要自己把这些破碎的文件合并起来如果你用UNIX或苹果,用 cat zhao.part* > zhao.MP3就可以如果用的是Windows,用copy /b 来解决吧,呵呵<br />
上面讲的都是http协议的下载,其实ftp也一样可以用。用法嘛,<br />
$ curl -u name:passwd ftp://ip:port/path/file<br />
或者大家熟悉的<br />
$ curl ftp://name:passwd@ip:port/path/file</p>
<ol>
<li>说完了下载,接下来自然该讲上传咯上传的option是 -T<br />
比如我们向ftp传一个文件:<br />
$ curl -T localfile -u name:passwd ftp://upload_site:port/path/<br />
当然,向http服务器上传文件也可以比如<br />
$ curl -T localfile http://cgi2.tky.3web.ne.jp/~zzh/abc.cgi<br />
注意,这时候,使用的协议是HTTP的PUT method</li>
</ol>
<p>刚才说到PUT,嘿嘿,自然让老服想起来了其他几种methos还没讲呢! GET和POST都不能忘哦。<br />
http提交一个表单,比较常用的是POST模式和GET模式<br />
GET模式什么option都不用,只需要把变量写在url里面就可以了比如:<br />
$ curl http://www.linuxidc.com/login.cgi?user=nickwolfe&password=12345</p>
<p>而POST模式的option则是 -d<br />
比如:<br />
$ curl -d “user=nickwolfe&password=12345” http://www.linuxidc.com/login.cgi<br />
就相当于向这个站点发出一次登陆申请~<br />
到底该用GET模式还是POST模式,要看对面服务器的程序设定。</p>
<p>一点需要注意的是,POST模式下的文件上的文件上传,比如<br />
```html</p>
<form method="POST" enctype="multipar/form-data" action="http://cgi2.tky.3web.ne.jp/~zzh/up_file.cgi">
<input type=submit name=nick value="go">
```
这样一个HTTP表单,我们要用curl进行模拟,就该是这样的语法:
$ curl -F upload=@localfile -F nick=go http://cgi2.tky.3web.ne.jp/~zzh/up_file.cgi
罗罗嗦嗦讲了这么多,其实curl还有很多很多技巧和用法比如 https的时候使用本地证书,就可以这样
$ curl -E localcert.pem https://remote_server
再比如,你还可以用curl通过dict协议去查字典~
$ curl dict://dict.org/d:computer
1. 开启gzip请求
curl -I http://www.sina.com.cn/ -H "Accept-Encoding:gzip,defalte"
2. 监控网页的响应时间
curl -o /dev/null -s -w "time_connect:%{time_connect}\ntime_starttransfer:%{time_starttransfer}\ntime_total:%{time_total}\n" http://www.kklinux.com
3. 监控站点可用性
curl -o /dev/null -s -w %{http_code} http://www.kklinux.com
</form>
awk调用shell变量
2009-11-03T00:00:00+08:00
bash
awk
http://chenlinux.com/2009/11/03/using-shell-variable-in-awk-script
<p>今天的问题:因为某个原因,需要长期探测对某机器的ping值情况。期望的输出格式是“丢包率 响应时间均值”。</p>
<p>写个小脚本,最后echo一下,自然好办的很。不过在crontab里看到之前大都有一条任务写的是ping 1.2.3.4,于是想:能不能让这个脚本的内容也尽量写在一句话里呢?</p>
<p>连动命令的话,输出结果都分了行。于是开始摸索awk的内外变量调用问题。</p>
<p>网上说明很多,大都是BEGIN或者-v的办法。一一试过后,发现其结果也都是分行显示的。</p>
<p>最后大海淘沙般找出了适用的写法。结果全在’“<code class="highlighter-rouge">的区分上——而且我至今不知道为啥非得按如下写法才行:
</code><code class="highlighter-rouge">bash
ls=`ping -c 5 1.2.3.4 | grep loss | awk -F, '{print $3}'`;ping -c 5 1.2.3.4 | grep rtt | awk -F/ '{print "'"$ls"' avg " $5 "ms"}'
</code><code class="highlighter-rouge">
执行显示结果如下:
0% packet loss avg 17.486ms
——————————————————————————————
时隔近月,在熟悉了awk的变量以后,我发现其实没有这么复杂,只要下面这样一句就简单搞定了:
</code><code class="highlighter-rouge">bash
ping -c 5 1.2.3.4|awk 'BEGIN{RS="##";FS=",|/"}{print $3,$5,$8“ms”}'
</code>`<br />
执行结果同上。</p>
squid压力测试
2009-11-03T00:00:00+08:00
testing
squid
http_load
http://chenlinux.com/2009/11/03/test-squid-by-http_load
<p>向公司申请了台设备做测试机,打算把公司各种应用服务都自己练练,熟悉一下。先从最传统的squid开始做压力测试。先发一个小东东http_load的测试:</p>
<ul>
<li>实验环境:</li>
</ul>
<p>服务器硬件条件:内存2096M,CPU2.33GHz,硬盘70G;<br />
Squid版本:Version 2.6.STABLE22</p>
<ul>
<li>实验架构:</li>
</ul>
<p>单台服务器:沈阳<br />
web页面由nginx发布,采用基础配置,监听8080端口,网站页面类型包括.htm/html/css/js/xml;<br />
前端由squid代理,采用公司默认配置,由cache_peer请求本机页面,监听80端口;<br />
测试访问地点:北京</p>
<ul>
<li>测试方法:</li>
</ul>
<p>创建url文件,添加url记录共11条,其中html六条,js三条,css/xml各一条。(本来还有一个tar.gz文件,结果文件有近2M,影响测试结果,放弃)<br />
根据网友经验,采取fetches参数配合parallel参数测试。</p>
<p>命令如下:./http_load -f 3000 -p 100 url > result.txt</p>
<p>根据情况调整parallel参数。</p>
<ul>
<li>测试结果:</li>
</ul>
<p>** 当p到180以上后,测试过程中就开始出现类似下面这样提示:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>http://www.test.com/ts.js: Connection timed out
http://www.test.com/ts.js: byte count wrong
</code></pre>
</div>
<p>而结果中的fetches/sec则在650-850之间随机出现,属于不太稳定的状态了。而p再往上提高(我从200一直实验到500), fetches/sec基本都在800左右,而msecs/connect则从50上升到255左右。<br />
** 当p在140以下时,fetches/sec在1500以上,msecs/connect在0.2左右,bytes/sec在85000左右(此时服务器iptraf -d eth1查看流量大概每秒12M);<br />
** 当p到140以上后,f/s迅速下降到1000左右,ms/c则上升到10以上;parallel从150到180的过程中,f/s、ms/c和b/s基本是均速变化的;同时查看服务器的iostat或者vmstat,一般us在90%多,bo偶然有1出现(后来发现是因为我放的测试文件大小相差较大)。</p>
<ul>
<li>实验结论</li>
</ul>
<p>squid服务在并发数140以下时,能够提供优质的服务,140以上,性能逐渐下降,当并发数到200以上后,遭遇瓶颈,主要在CPU方面。</p>
<ul>
<li>实验疑问</li>
</ul>
<p>公司在线服务的设备,一般eth1流量都能跑到60M,有些甚至上了100M,这跟测试结果相差也太大了?</p>
<ul>
<li>参考资料<br />
<a href="http://www.hiadmin.com/tomcat-%e5%b9%b6%e5%8f%91%e6%b5%8b%e8%af%95/">http://www.hiadmin.com/tomcat-%e5%b9%b6%e5%8f%91%e6%b5%8b%e8%af%95/</a><br />
./http_load -parallel 200 -seconds 10 urls<br />
按照固定时间来结束测试,这样可以比较相同时间内被测服务器的响应速度.<br />
./http_load -parallel 200 -fetches 1000 urls<br />
按照固定申请数来测试,这样可以比较相同访问量下返回的响应速度.<br />
虽然两者都可以获取到服务器的响应速度<br />
但是使用fetches更容易让被测服务器收到压力<br />
由于seconds控制测试时间,很有可能在短时间内测试客户端并没有发起足够数量的请求<br />
而服务端在收到足够压力之前,测试就已经结束了.<br />
有一些情况,诸如内存泄漏以及资源回收不利或者对后面的响应速度越来越慢等情况<br />
在这种测试条件下不容易发生<br />
而使用fetchs,能够让客户端保证确定请求数的全部处理.<br />
使用时间作为控制参数<br />
会由于测试人员不够耐心而人为将seconds参数设置过小<br />
导致测试结果失去意义<br />
所以,最后建议使用fetches作为测试参数.用以作为基准进行比较<br />
如果httpd_load获取到的页面数据和上次不一致<br />
则会报错byte count wrong<br />
如果是动态页面,由于返回数据内容不同.则此报错可以忽略</li>
</ul>
tcpwrapper
2009-11-03T00:00:00+08:00
linux
http://chenlinux.com/2009/11/03/tcpwrapper
<p>今天去新浪面试。有一道笔试题,考的是tcpwrapper的用法。因为没见过这个东东,所以百度一下,选几篇文章总结一下。</p>
<p>首先,tcpwrapper是unix上的工具,1990年就诞生了。至于它和iptables的不同,看到有人说是TCP/IP层的不同。说iptables是网络层的,tcpwrapper是应用层的。对不对,且看tcpwrapper的使用先:</p>
<ol>
<li>
<p>部署:首选当然是用安装包,要是编译源码,参见如下文:http://echo.sharera.com/blog/BlogTopic/9379.htm,虽然作者说自己笨笨的乱写,那也比我强多了</p>
</li>
<li>
<p>开启日志:在/etc/syslog.conf里添加如下字段即可</p>
</li>
</ol>
<p>tcpwrapper loglocal3.info /var/log/tcplog</p>
<p>这个时候要记得重启日志服务。可以使用kill -HUP syslogd进程号的方法(nnd,这也是今天的笔试题之一)。</p>
<ol>
<li>配置文件:/etc/hosts.allow</li>
</ol>
<p>(本来还有个hosts.deny的)<br />
编写规则是“servicename:hostname[:shellcmd]”</p>
<p>tcpwrapper监控的是inetd里的启动服务,用telnet举例如下</p>
<div class="highlighter-rouge"><pre class="highlight"><code>telnet:ALL
EXCEPT LOCAL, .M-gtuiw.com
<span class="nb">echo</span> <span class="s2">"request from %d@%h:"</span> >> /var/log/telnet.log;
<span class="k">if</span> <span class="o">[</span> %h !<span class="o">=</span> <span class="s2">"OS.M-gtuiw.com:"</span> <span class="o">]</span> ; <span class="k">then
</span>finge -l @%h >> /var/log/telnet.log
<span class="k">fi</span>
</code></pre>
</div>
<p>意即允许除了本机和M-gtuiw.com域下主机以外的所有telnet请求,并以“请求来自服务名@主机名”的方式记录进日志。(注意:EXCEPT也可以用在servicename后面)</p>
<p>和iptables一样(好像说反了,其实应该是iptables和tcpd一样),这个allow和deny的规则也是讲究先来后到的,所以会有个ALL:ALL:deny收尾(如果单有deny文件,就在里头写ALL:ALL就可以了)。</p>
<ol>
<li>调试</li>
</ol>
<p>tcpdchk -v可以看到tcpd的全部规则设置和错误提示<br />
tcpdmatch servicename hostname可以具体查询某条规则</p>
<ol>
<li>inetd服务配置</li>
</ol>
<p>相关的有两个文件,一个是/etc/services,这里定义了各种服务使用的协议和占用的端口(本文中出现的第三个新浪笔试考题了哦~~);一个是/etc/inetd.conf,这里定义了各项服务的类型、协议、监听方式、用户、程序、参数——如果启用tcpwrapper的话,程序就都是/usr/sbin/tcpd了。比如telnet的配置行如下:</p>
<p>telnet stream tcp nowait root /usr/sbin/tcpd in.telnetd</p>
<ol>
<li>日志结果,直接摘抄一段如下:</li>
</ol>
<div class="highlighter-rouge"><pre class="highlight"><code>Jul 31 22:00:52 <span class="o">[</span>url]www.test.org[/url] <span class="k">in</span>.telnetd[4365]: connect from 10.68.32.1
Jul 31 22:02:10 <span class="o">[</span>url]www.test.org[/url] <span class="k">in</span>.telnetd[4389]: connect from 10.68.32.5
Jul 31 22:04:58 <span class="o">[</span>url]www.test.org[/url] <span class="k">in</span>.ftpd[4429]: connect from 10.68.32.3
</code></pre>
</div>
<p>以上说了这么多,都是unix上的,最后来一句,在linux上,xinetd就是这个inetd+tcpwrapper了。何况还有强大的iptables……它可不像tcpwrapper只能管tcp协议的服务哦~~</p>
<p>参考文章:</p>
<p><a href="http://jianjian.blog.51cto.com/35031/41949">http://jianjian.blog.51cto.com/35031/41949</a><br />
<a href="http://echo.sharera.com/blog/BlogTopic/9379.htm">http://echo.sharera.com/blog/BlogTopic/9379.htm</a><br />
<a href="http://blog.chinaunix.net/u/26264/showart_971334.html">http://blog.chinaunix.net/u/26264/showart_971334.html</a><br />
http://www.linuxdiyf.com/viewarticle.php?id=18335</p>
shell处理技巧
2009-11-03T00:00:00+08:00
bash
http://chenlinux.com/2009/11/03/some-useful-shell-techniques
<p>脚本经常用的上,但老是记不住的一些东西:</p>
<p>一、逻辑运算</p>
<div class="highlighter-rouge"><pre class="highlight"><code>eq 相等
ne、neq 不相等
gt 大于
lt 小于
gte、ge 大于等于
lte、le 小于等于
not 非
mod 求模
is [not] div by 是否能被某数整除
is [not] even 是否为偶数
is [not] odd 是否为奇数
-a 存在则为真
-b 存在且是一个块特殊文件则为真
-c 存在且是一个字特殊文件则为真
-d 存在且是一个目录则为真
-e 存在则为真
-f 存在且是一个普通文件则为真
-g 存在且已经设置了SGID则为真
-h 存在且是一个符号连接则为真
-k 存在且已经设置了粘制位则为真
-p 存在且是一个名字管道(F如果O)则为真
-r 存在且是可读的则为真
-s 存在且大小不为0则为真
-t 打开且指向一个终端则为真
-u 存在且设置了SUID则为真
-w 存在且是可写的则为真
-x 存在且是可执行的则为真
-O 存在且属有效用户ID则为真
-G 存在且属有效用户组则为真
-L 存在且是一个符号连接则为真
-N 存在 and has been mod如果ied since it was last
read则为真
-S 存在且是一个套接字则为真
</code></pre>
</div>
<p>二、特殊变量</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$# 传递到脚本的参数个数
$* 以一个单字符串显示所有向脚本传递的参数。与位置变量不同,此选项参数可超过9个
$$ 脚本运行的当前进程ID号
$! 后台运行的最后一个进程的进程ID号
$@ 与$#相同,但是使用时加引号,并在引号中返回每个参数
$- 显示shell使用的当前选项,与set命令功能相同
$? 显示最后命令的退出状态。0表示没有错误,其他任何值表明有错误
$((...)) 对括号内的表达式求值
</code></pre>
</div>
<p>三、打印格式</p>
<table>
<tbody>
<tr>
<td>free -m</td>
<td>awk ‘/Mem/{printf “%.0fn”,$1*0.9}’</td>
</tr>
</tbody>
</table>
<p>自动取整数: %代表任意长度,0表示保留.后0位数字</p>
<p>四、翻转文件</p>
<p>sed ‘1!G;h;$!d’ file<br />
sed -n ‘1!G;h;$p’ file</p>
<p>sed的man说法如下:<br />
h<br />
H Copy/append<br />
pattern space to hold space.<br />
g<br />
G Copy/append<br />
hold space to pattern space.<br />
这两个space,我的理解就是hold是厂房(原料和成品都放这,汗~),pattern是流水线。 <br />
如果是正常的sed,应该是所有的原料一起上流水线处理,然后统一成品输出。所以顺序不变;但上面这个命令,每处理一行就清空一次流水线,最后成品就成倒序了。<br />
还有一个x,用来互换两个space的文本。</p>
<p>五、正则表达式</p>
<div class="highlighter-rouge"><pre class="highlight"><code>pattern{n}:只用来匹配前面pattern出现的次数.n为次数。如a{2}匹配aa.
pattern{n,}:含义同上,但次数最少为n.如a{2,}匹配aa,aaa,aaaa,.....
pattern{n,m}:含义同上,但次数在n和m之间。如a{2,4}匹配aa,aaa,aaaa三个
[0123456789]或[0-9] :假定要匹配任意一个数字
[a-z] :任意小写字母
[A-Za-z] :任意大小写字母
[S,s] :匹配大小写S
[0-9]{3}.[0-9]{3}.[0-9]{3}.[0-9]{3} :匹配IP地址 [0-9]{3}三个0-9组成的字符串;. :匹配点(注意这里点是特殊的字符,所以要用""来屏蔽其含义)
用于grep和awk的类名(其他能不能我还不知道)
[[:upper:]] 表示[A-Z]
[[:alnum:]] 表示[0-9a-zA-Z]
[[:lower:]] 表示[a-z]
[[:space:]] 表示空格或者tab键
[[:digit:]] 表示[0-9]
[[:alpha:]] 表示[a-zA-Z]
</code></pre>
</div>
squid问题-域名解析
2009-11-03T00:00:00+08:00
squid
http://chenlinux.com/2009/11/03/priority-of-squid-domain-resolve
<p>今天有老客户下单修改源IP地址1.2.3.4为1.2.3.7,一切正常操作过后进行测试,其中有台机器就是狂报404。<br />
在用/home/squid/bin/squidclient -p 80 -m PURGE http://测试url<br />
命令清除缓存,甚至重启dns/squid服务后,其测试访问的first-to-parent地址还是1.2.3.4!!</p>
<p>用dig检查确认内部DNS配置已经生效后,又检查了hosts文件也没有问题。</p>
<p>最后发现是squid.conf里的泛域名配置问题。</p>
<p>这批服务器在升级squid前,曾经在一台机器上测试新版本配置,之后一直没有更改,其中有如下字段:</p>
<pre><code class="language-squid">cache_peer 1.2.3.4 parent 80 0 no-query no-netdb-exchange originserver
cache_peer_domain 1.2.3.4 www.test.com
</code></pre>
<p>所以在系统上怎么修改,都没法成功了。<br />
由此可知,CDN加速对域名的解析,是squid配置文件最优先,然后才是系统的hosts文件,最后是DNS服务器。</p>
修改dbname常见的一个错误NID-00135及解决…
2009-11-03T00:00:00+08:00
database
oracle
http://chenlinux.com/2009/11/03/nid-00135-of-dbname
<p>oracle自带有nid用以修改dbname。查看其命令语法如下表所示:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>DBNEWID: Release 10.2.0.4.0 - Production on Wed Jun 24 20:06:08 2009
Copyright (c) 1982, 2007,
Oracle. All
rights reserved.
Keyword
Description (Default)
----------------------------------------------------
TARGET Username/Password (NONE)
DBNAME New
database name (NONE)
LOGFILE
Output Log
(NONE)
REVERT Revert
failed
change
NO
SETNAME
Set a new database name
only
NO
APPEND Append
to output log
NO
HELP Displays
these
messages NO 其动作描述用“=”表示。
</code></pre>
</div>
<p>假设某oracle数据库sys密码为123456,欲更名dbname为aaa,则其修改dbname的命令应如下行所示:</p>
<p>nid target=sys/123456 dbname=aaa</p>
<p>网上关于修改dbname的博客文章和论坛问答,基本都是在windows平台上的操作。其提示要点,在于运行nid系统命令之前,必须将数据库置于mount状态下。以此类推,在linux下的操作步骤,应该如下:<br />
<code class="highlighter-rouge">sql
sql > shutdown
immediate;
sql > startup mount;
sql > host nid target=sys/123456 dbname=aaa
</code><br />
但我按此步骤进行之后,却提示如下字段:<br />
NID-00135: There are 1 active threads<br />
Change of database name failed during validation - database is<br />
intact.<br />
DBNEWID - Completed with validation errors.<br />
经过多机试验,发现这个错误并非偶然出现一两次而已,至于windows平台下,为何无人提起,就有待日后研究了。<br />
关于这个错误,关键是检查两个地方。<br />
第一是表空间与数据文件的状态:<br />
SQL> select<br />
file#,status,name from<br />
v$datafile;<br />
FILE#<br />
STATUS NAME<br />
———- ——-<br />
——————————————————————————–<br />
1<br />
SYSTEM /u01/app/oracle/oradata/db1/system01.dbf<br />
2<br />
ONLINE /u01/app/oracle/oradata/db1/undotbs01.dbf<br />
3<br />
ONLINE /u01/app/oracle/oradata/db1/sysaux01.dbf<br />
4<br />
ONLINE /u01/app/oracle/oradata/db1/usertbs.dbf<br />
5<br />
ONLINE /u01/app/oracle/oradata/db1/raocl.dbf<br />
正常情况下,其状态应该是online或者offline。但如果因为历史操作的原因,导致某数据文件的状态变成了recovery,那么就会出问题了。<br />
解决方法也简单,drop掉出错的数据文件就行了。<br />
第二是归档文件的设置:<br />
SQL> archive log<br />
list<br />
Database log<br />
mode Archive<br />
Mode<br />
Automatic<br />
archival<br />
Enabled<br />
Archive<br />
destination /u01/app/oracle/product/10.2.0/db1/dbs/arch<br />
Oldest online log<br />
sequence<br />
31<br />
Next log sequence to<br />
archive<br />
33<br />
Current log<br />
sequence<br />
33<br />
SQL> host ls $ORACLE_HOME/dbs<br />
alert_db1.log<br />
arch1_29_689269707.dbf control01.ctl<br />
initdw.ora spfiledb1.ora.bak<br />
arch1_25_689269707.dbf arch1_30_689269707.dbf control02.ctl<br />
init.ora<br />
arch1_26_689269707.dbf arch1_31_689269707.dbf db1_ora_4704.trc lkAAA<br />
arch1_27_689269707.dbf arch1_32_689269707.dbf hc_db1.dat lkDB1<br />
arch1_28_689269707.dbf cntrldb1.dbf initdb1.ora<br />
orapwdb1<br />
如果没有设置归档文件路径或者没有归档文件存在,nid也会出错。<br />
设置归档文件模式、路径并手工归档的命令分别如下:<br />
SQL> alter database archivelog;<br />
SQL> alter system<br />
set log_archive_dest_1=’location=/u01/app/oracle/oradata/db1/arch’;<br />
SQL> alter system<br />
archive log current;<br />
注意:归档文件模式也要在mount下设置。<br />
确认完成这两步以后,在重新运行nid系统命令,出现如下字段,即可成功更改dbname了。<br />
Control Files in database:<br />
/u01/app/oracle/product/10.2.0/db1/dbs/control01.ctl<br />
/u01/app/oracle/product/10.2.0/db1/dbs/control02.ctl<br />
Change database ID and database<br />
name DB1 to AAA? (Y/[N]) => Y<br />
Proceeding with operation<br />
Changing database ID from 1283133323 to<br />
1845742016<br />
Changing database name from DB1<br />
to AAA<br />
Control<br />
File<br />
/u01/app/oracle/product/10.2.0/db1/dbs/control01.ctl -<br />
modified<br />
Control<br />
File<br />
/u01/app/oracle/product/10.2.0/db1/dbs/control02.ctl -<br />
modified<br />
Datafile<br />
/u01/app/oracle/oradata/db1/system01.dbf - dbid changed, wrote new<br />
name<br />
Datafile<br />
/u01/app/oracle/oradata/db1/undotbs01.dbf - dbid changed, wrote new<br />
name<br />
Datafile<br />
/u01/app/oracle/oradata/db1/sysaux01.dbf - dbid changed, wrote new<br />
name<br />
Datafile<br />
/u01/app/oracle/oradata/db1/usertbs.dbf - dbid changed, wrote new<br />
name<br />
Datafile<br />
/u01/app/oracle/oradata/db1/raocl.dbf - dbid changed, wrote new<br />
name<br />
Datafile<br />
/u01/app/oracle/oradata/db1/temp01.dbf - dbid changed, wrote new<br />
name<br />
Control<br />
File<br />
/u01/app/oracle/product/10.2.0/db1/dbs/control01.ctl - dbid<br />
changed, wrote new name<br />
Control<br />
File<br />
/u01/app/oracle/product/10.2.0/db1/dbs/control02.ctl - dbid<br />
changed, wrote new name<br />
Instance<br />
shut down<br />
Database name changed to<br />
AAA.<br />
Modify parameter file and generate a new password file before restarting.<br />
Database ID for database AAA<br />
changed to 1845742016.<br />
All previous backups and archived redo logs for this database are<br />
unusable.<br />
Database has been shutdown, open<br />
database with RESETLOGS option.<br />
Succesfully changed database<br />
name and<br />
ID.<br />
DBNEWID - Completed succesfully.<br />
至于引起这个错误的深层次原因,从之前有过的其他操作猜测,会不会是scn不一致的原因??如果是这个原因,那或许只要很简单的CKPT就可以了。找时间试验一下。</p>
sed使用
2009-11-03T00:00:00+08:00
bash
sed
http://chenlinux.com/2009/11/03/learning-sed
<p>shell这东西,只有活逼到手头上了,才知道应该去学什么。<br />
比如有客户要求修改/home/squid/share/errors/Simplify_Chinese/里所有的错误页面转向到他们自己专用的界面去,七八个节点,几十台服务器,几百个文件,一个一个vi编辑,多可怕。只好现学现卖sed,目前发现两种办法:</p>
<p>1、find+sed<br />
比如这样:<br />
find . -name “*.html” -exec sed -i “s/eht/the/g” {} ;<br />
用exec传输find的结果给sed,{}是集合的意思;</p>
<p>2、sed+grep<br />
比如这样:<br />
sed -i “s/eht/the/g” <code class="highlighter-rouge">grep eht -rl /test</code><br />
这个-rl参数,有时间研究一下。</p>
<p>或者这样:<br />
grep “abc” * -R | awk -F: ‘{print $1}’ | sort | uniq | xargs sed -i ‘s/abc/abcde/g’</p>
<p>awk -F:的意思是指定:为列的分隔符,sort排序,uniq删除重复行,最后用xargs传输大量数据给sed。</p>
<table>
<tbody>
<tr>
<td>说到xargs,比如曾经有一次/var/spool/clientmqueue目录占用了大量磁盘空间,但其中的文件都是4.0K,数量及其多,单纯用rm,无法达到目的,就得用ls</td>
<td>xargs rm -f才行。</td>
</tr>
</tbody>
</table>
expect脚本——批量修改ssh配置
2009-11-03T00:00:00+08:00
bash
ssh
http://chenlinux.com/2009/11/03/expect-script-to-modify-ssh-configurations-of-cluster
<p>公司服务器一般通过ssh进行远程管理。以前大家登录的时候,都是随意选内外网IP进入。王总接手后,说这事隐患太大了,必须禁了外网ssh。第一思路,用iptables把外网ssh的包DROP掉;第二思路,用tcpwrapper把sshd的allow写死;第三思路,修改sshd_config,只监听内网请求。</p>
<p>由于一些说不清楚的原因,iptables的办法没法用;而tcpwrapper占用CPU资源较多;所以最后决定用第三种办法。</p>
<p>公司服务器比较多,而且根据随机登录查看的结果,sshd_config内容居然还太不一样~~手工干了一天,改了两组服务器后,终于下定决心要整个全自动脚本出来干活……<br />
目前的办法是这样的:</p>
<p>cat ssh.exp<br />
<code class="highlighter-rouge">bash
#!/usr/bin/expect -f
log_file exp.log
set timeout -1
set ipaddr [lrange $argv 0 0]
for {set i 1} {$i<4} {incr i} {
spawn ssh $ipaddr
expect {
"*password:" break
"to host" {sleep 2};
sleep 3
}
}
send "123456r"
expect "]#"
send "cd /etc/sshr"
send "cp sshd_config sshd_config.`date +%F-%T`.bakr"
send "sed -i /^ListenAddress.*$/d sshd_configr"
send "echo ListenAddress `/sbin/ifconfig eth0|awk '/inet /{print $2}'|awk -F: '{print $2}'` >> sshd_configr"
send "service sshd restartr"
send "exitr"
interact
</code><br />
cat do.sh<br />
<code class="highlighter-rouge">bash
#!/bin/sh
for ip in `cat ip.lst`
do
./ssh.exp $ip > /dev/null 2>&1
done
cat exp.log | grep host | awk '{print $5}'|sort|uniq >> errorip
echo "以下IP无法修改";cat errorip
</code></p>
expect日志分析
2009-11-03T00:00:00+08:00
bash
expect
http://chenlinux.com/2009/11/03/expect-log-analysis
<p>接着上次expect脚本那事儿往下走。<br />
由于不同服务的管理方法不同,上次关闭了ssh的外网登录以后,各地不断有服务器报出这样那样的问题。主管一发狠:“全面检查!”<br />
在检查中,还真发现不少问题。最突出的就是很多本应该上传到中心服务器的日志居然一直留在本机没动弹!时不时发作出来,就撑爆了根分区——这当然有分区规划不合理的问题。但在线业务,磁盘划分修改起来就不是那么方便了。于是退而求其次,定期监控日志文件大小吧。这回expect只要du<br />
-sh一下就行了,方便的很。问题在下一步的分析。<br />
摘举exp.log中一次循环的执行结果如下:<br />
<code class="highlighter-rouge">bash
The authenticity of host '1.2.3.4 (1.2.3.4)' can't be established.
RSA key fingerprint is
bb:d5:81:e1:84:09:c5:32:f6:fb:e1:b3:d3:de:c3:53.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '1.2.3.4' (RSA) to the list of known hosts.
root@1.2.3.4's password:
4.0K /home/apache2/logs/access_log
</code><br />
第一步,是用如下脚本,可以做到提取日志达到50M大小的服务器IP。<br />
<code class="highlighter-rouge">bash
#!/bin/bash
nk=`sed -n -e "/50M/=" exp.log`
nnk=`expr $nk - 1`
sed -n "$nnk"p"" exp.log|awk -F"'" '{print $1}'|awk -F"@" '{print $2}'
</code><br />
但问题是:如果同时有两台到50M呢?或者在运行到它时,已经到50M以上呢?<br />
于是我想,以ls -sh显示大小,人眼好看,电脑不好认啊。如果用du -b,那大小相同的几率就应该小很多很多了。然后定一个阀值,进行比较循环就可以了。<br />
最后脚本如下:<br />
<code class="highlighter-rouge">bash
#!/bin/bash
for ip in `cat ip.lst`
do
./ssh.exp $ip > /dev/null 2&>1
done
bs=400
size=`grep access exp.log | awk '{if ($1&gt;'"$bs"'){print $1}}'`
for so in $size
do
nk=`sed -n -e "/$so/=" exp.log`
nnk=`expr $nk - 1`
sed -n "$nnk"p"" exp.log | awk -F"'" '{print $1}'|awk -F@ '{print $2}'
done
</code><br />
#本来想用sed ‘N;s/n//g’ exp.log来合并行尾,省得调行号,但exp日志的格式因为ssh登录的提示信息不一而无法统一,只能放弃。<br />
试验性的在ip.lst中输入了15个IP,运行结果显示出来了两个。成功。</p>
awk变量$0妙用
2009-11-03T00:00:00+08:00
bash
awk
http://chenlinux.com/2009/11/03/dollar-zero-in-awk
<p>接着说上篇的脚本。<br />
因为看awk看的入迷,边想把exp.log的处理那段都用awk写出来。惊喜的发现awk有个内置参数NR,而且awk内部也可以进行运算。<br />
于是上次的脚本就改成了这个样子:<br />
<code class="highlighter-rouge">bash
#!/bin/bash
for ip in `cat ip.lst`
do
./ssh.exp $ip > /dev/null 2&>1
done
NK=`awk 'BEGIN{bs=4000000}/access/{if($1>bs){nk=NR-1;print nk}}' exp.log`
for nnk in $NK
do
awk -F"[@|']" 'NR=='"$nnk"' {print $2}' exp.log
done
</code><br />
然后又发现awk中$0的鬼怪。于是进一步简化成了这个样子:<br />
<code class="highlighter-rouge">bash
#!/bin/bash
for ip in
`cat ip.lst`
do
./ssh.exp $ip > /dev/null 2&>1
done
awk 'BEGIN{bs=4000000}/access/{if($1>bs)print x};{x=$0}' exp.log|awk -F"[@|']" '{print $2}'
</code><br />
终于算是圆了自己用一句话搞定它的梦。yeah~<br />
不过对这个原理还是不很明白。因为print x;x=$0出来是上一行,但print $0则是本行。why?<br />
网上对打印前一行还提出另一个写法,就看的更莫名其妙了:<br />
<code class="highlighter-rouge">bash
awk '/regex/{print (x==""?"":x)};{x=$0}' $1
</code><br />
而打印后一行是这样:<br />
<code class="highlighter-rouge">bash
awk '/regex/{getline;print}' $1
</code><br />
不过这毕竟是恰好上下行而已,如果是要前几行的,还是要靠NR运算了。</p>
<hr />
<p>这个问题现在已经知道了,因为awk的流式处理,print x;x=$0,这个时候的x要等到下一行时才print出来。</p>
squid 自动配置脚本
2009-11-03T00:00:00+08:00
devops
squid
web
bash
http://chenlinux.com/2009/11/03/automate-configure-squid
<p>公司刚重新规定了squid服务的标准配置。以后,客户的配置要求,统一安排。这样,除了一些有特殊要求(比如加密/防盗链等)的以外,普通客户就可以逐步实现简洁明了的规范化自动化配置。</p>
<p>目前squid的初始化准备还在慢慢进行中,趁有点空闲,先把自动化配置脚本搞出来。</p>
<p>主程序do.sh如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="c">#!/bin/bash</span>
<span class="k">if</span> <span class="o">[</span> <span class="s2">"$#"</span> -ne 2 <span class="o">]</span>;<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Usage: ./do.sh [command][customer]"</span>
<span class="nb">echo</span> <span class="s2">"For example:./do.sh add abc"</span>
<span class="nb">echo</span> <span class="s2">" ./do.sh del abc"</span>
<span class="nb">exit </span>1
<span class="k">elif</span> <span class="o">[</span> ! -s ip.lst -a -e ip.lst <span class="o">]</span>;<span class="k">then
</span><span class="nb">echo</span> <span class="s2">"No server in ip.lst"</span>
<span class="nb">exit </span>1
<span class="k">fi
for </span>ip <span class="k">in</span> <span class="sb">`</span>cat ip.lst<span class="sb">`</span>;<span class="k">do
</span>ping -c 5 <span class="nv">$ip</span>
expect ssh.exp <span class="nv">$ip</span> <span class="nv">$1</span> <span class="nv">$2</span>
<span class="k">done</span>
</code></pre>
</div>
<p>ssh.exp如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> #!/usr/bin/expect -f
log_file exp.log
set ipaddr <span class="p">[</span><span class="nb">lindex</span> <span class="nv">$argv</span> 0<span class="p">]</span>
set command <span class="p">[</span><span class="nb">lindex</span> <span class="nv">$argv</span> 1<span class="p">]</span>
set custom <span class="p">[</span><span class="nb">lindex</span> <span class="nv">$argv</span> 2<span class="p">]</span>
spawn scp /squid.config/$custom /rt/conf.sh <span class="nv">$ipaddr:</span>/root/
for <span class="p">{}</span> 1 <span class="p">{}</span> <span class="p">{</span>
expect <span class="p">{</span>
eof
break
<span class="s2">"(yes/no)?"</span> <span class="p">{</span>send <span class="s2">"yesr"</span><span class="p">}</span>
<span class="s2">"assword:"</span> <span class="p">{</span>send <span class="s2">"123456r"</span><span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
spawn ssh root@$ipaddr bash conf.sh <span class="nv">$command</span> <span class="nv">$custom</span>
for <span class="p">{}</span> 1 <span class="p">{}</span> <span class="p">{</span>
expect <span class="p">{</span>
eof
break
<span class="s2">"(yes/no)?"</span> <span class="p">{</span>send <span class="s2">"yesr"</span><span class="p">}</span>
<span class="s2">"assword:"</span> <span class="p">{</span>send <span class="s2">"123456r"</span><span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>
<p>conf.sh如下:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <span class="c">#!/bin/bash</span>
<span class="nv">NR</span><span class="o">=</span><span class="k">$(</span>cat <span class="nv">$2</span>|wc -l<span class="k">)</span>
<span class="nv">CONF</span><span class="o">=</span>/etc/squid.conf
<span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"add"</span> <span class="o">]</span>;<span class="k">then
</span>sed -i <span class="s2">"/config/r </span><span class="nv">$2</span><span class="s2">"</span> <span class="nv">$CONF</span>
<span class="k">elif</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"del"</span> <span class="o">]</span>;<span class="k">then
</span><span class="nv">CFNR</span><span class="o">=</span><span class="sb">`</span>sed -n -e /<span class="nv">$2</span>/<span class="o">=</span> <span class="nv">$CONF</span><span class="sb">`</span>
<span class="nv">BEGINNR</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="nv">$CFNR</span>|awk <span class="s1">'{print $1}'</span><span class="sb">`</span>
<span class="nv">ENDNR</span><span class="o">=</span><span class="sb">`</span>expr <span class="nv">$BEGINNR</span> + <span class="nv">$NR</span><span class="sb">`</span>
sed -i <span class="s2">"/</span><span class="nv">$BEGINNR</span><span class="s2">/,/</span><span class="nv">$ENDNR</span><span class="s2">/d"</span> <span class="nv">$CONF</span>
rm -f <span class="s2">"</span><span class="nv">$2</span><span class="s2">"</span><span class="k">*</span>
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"Use add or del please!"</span>
<span class="k">fi
</span>killall -9 squid
<span class="nb">ulimit</span> -HSn 655360
/sbin/squid -s
</code></pre>
</div>
<p>然后我们模拟一个叫做abc的客户来测试CDN了。那么我只要在/squid.config/下创建一个叫做abc的文本,内容是针对性的配置部分字段,假设如下:</p>
<pre><code class="language-squid"> #abc
refresh_pattern -i http://www.abc.com/.*.(jpg|gif|js|css|swf|xml)$ 1440 50% 4320 ignore-reload
acl abc url_regex -i ^http://www.abc.com/.*.(html|do|jsp|asp|aspx|axd|asmx)
cache deny abc
</code></pre>
<p>然后运行./do.sh add abc,就可以自动在ip.lst里所有的服务器的squid.conf中的“#config”字段下面,添上abc的配置文件了。</p>
<p>过一段时间,要是abc这个客户流失了,就运行./do.sh del abc就行了。</p>
<p>之前的思路,局限在一句句往配置文件里插句子。于是用下面这个办法</p>
<div class="highlighter-rouge"><pre class="highlight"><code> sed -i s/<span class="s2">"config"</span>/<span class="o">{</span>
a<span class="s2">"……"</span>/
a<span class="s2">"……"</span>/
a<span class="s2">"……"</span>/
<span class="o">}</span>
squid.conf
</code></pre>
</div>
<p>而这样配置句段的顺序就反过来了,还得用 <code class="highlighter-rouge">sed -n "1!G;h;$!d"</code> 命令倒序读取——最开始用cat命令,结果cat在读取abc这个文件的时候会自动把空格前后的内容分段读出,于是改用sed。至于倒序之后,再怎么插入,就没有研究了。因为当时我发现了可以直接将文件a插入文件b的方法~~</p>
<table>
<tbody>
<tr>
<td>del的时候,其实在操作上还有一个办法。就是每次的配置不单以#abc开头,还用一个#abcEND结尾。这样,ENDNR就不用计算,直接取echo $CFNR</td>
<td>awk ‘{print $NF}’就行了。</td>
</tr>
</tbody>
</table>
<p>说到这个CFNR,让人很郁闷的一点是,这个sed -n -e出来的几个数字,我本意是作一个数组的,但怎么试验,shell都把这一串数字存在${CFNR[0]}一个元素里……数组到底怎么输入,还得研究~~</p>