1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
|
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>email_reply_parser.rb</title>
<link rel="stylesheet" href="http://github.com/jashkenas/docco/raw/0.3.0/resources/docco.css">
</head>
<body>
<div id='container'>
<div id="background"></div>
<table cellspacing=0 cellpadding=0>
<thead>
<tr>
<th class=docs><h1>email_reply_parser.rb</h1></th>
<th class=code></th>
</tr>
</thead>
<tbody>
<tr id='section-1'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-1">¶</a>
</div>
</td>
<td class=code>
<div class='highlight'><pre><span class="nb">require</span> <span class="s1">'strscan'</span></pre></div>
</td>
</tr>
<tr id='section-2'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-2">¶</a>
</div>
<p>EmailReplyParser is a small library to parse plain text email content. The
goal is to identify which fragments are quoted, part of a signature, or
original body content. We want to support both top and bottom posters, so
no simple “REPLY ABOVE HERE” content is used.</p>
<p>Beyond RFC 5322 (which is handled by the <a href="https://github.com/mikel/mail">Ruby mail gem</a>), there aren’t
any real standards for how emails are created. This attempts to parse out
common conventions for things like replies:</p>
<pre><code>this is some text
On <date>, <author> wrote:
> blah blah
> blah blah
</code></pre>
<p>… and signatures:</p>
<pre><code>this is some text
--
Bob
http://homepage.com/~bob
</code></pre>
<p>Each of these are parsed into Fragment objects.</p>
<p>EmailReplyParser also attempts to figure out which of these blocks should
be hidden from users.</p>
</td>
<td class=code>
<div class='highlight'><pre><span class="k">class</span> <span class="nc">EmailReplyParser</span>
<span class="no">VERSION</span> <span class="o">=</span> <span class="s2">"0.2.0"</span></pre></div>
</td>
</tr>
<tr id='section-3'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-3">¶</a>
</div>
<p>Splits an email body into a list of Fragments.</p>
<p>text – A String email body.</p>
<p>Returns an Email instance.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">read</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
<span class="no">Email</span><span class="o">.</span><span class="n">new</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
<span class="k">end</span></pre></div>
</td>
</tr>
<tr id='section-Emails'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-Emails">¶</a>
</div>
<h2>Emails</h2>
</td>
<td class=code>
<div class='highlight'><pre></pre></div>
</td>
</tr>
<tr id='section-5'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-5">¶</a>
</div>
<p>An Email instance represents a parsed body String.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="k">class</span> <span class="nc">Email</span></pre></div>
</td>
</tr>
<tr id='section-6'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-6">¶</a>
</div>
<p>Emails have an Array of Fragments.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="kp">attr_reader</span> <span class="ss">:fragments</span>
<span class="k">def</span> <span class="nf">initialize</span>
<span class="vi">@fragments</span> <span class="o">=</span> <span class="o">[]</span>
<span class="k">end</span></pre></div>
</td>
</tr>
<tr id='section-7'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-7">¶</a>
</div>
<p>Splits the given text into a list of Fragments. This is roughly done by
reversing the text and parsing from the bottom to the top. This way we
can check for ‘On <date>, <author> wrote:’ lines above quoted blocks.</p>
<p>text – A String email body.</p>
<p>Returns this same Email instance.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="k">def</span> <span class="nf">read</span><span class="p">(</span><span class="n">text</span><span class="p">)</span></pre></div>
</td>
</tr>
<tr id='section-8'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-8">¶</a>
</div>
<p>The text is reversed initially due to the way we check for hidden
fragments.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="n">text</span><span class="o">.</span><span class="n">reverse!</span></pre></div>
</td>
</tr>
<tr id='section-9'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-9">¶</a>
</div>
<p>This determines if any ‘visible’ Fragment has been found. Once any
visible Fragment is found, stop looking for hidden ones.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="vi">@found_visible</span> <span class="o">=</span> <span class="kp">false</span></pre></div>
</td>
</tr>
<tr id='section-10'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-10">¶</a>
</div>
<p>This instance variable points to the current Fragment. If the matched
line fits, it should be added to this Fragment. Otherwise, finish it
and start a new Fragment.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="vi">@fragment</span> <span class="o">=</span> <span class="kp">nil</span></pre></div>
</td>
</tr>
<tr id='section-11'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-11">¶</a>
</div>
<p>Use the StringScanner to pull out each line of the email content.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="vi">@scanner</span> <span class="o">=</span> <span class="no">StringScanner</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
<span class="k">while</span> <span class="n">line</span> <span class="o">=</span> <span class="vi">@scanner</span><span class="o">.</span><span class="n">scan_until</span><span class="p">(</span><span class="sr">/\n/</span><span class="p">)</span>
<span class="n">scan_line</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
<span class="k">end</span></pre></div>
</td>
</tr>
<tr id='section-12'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-12">¶</a>
</div>
<p>Be sure to parse the last line of the email.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="k">if</span> <span class="p">(</span><span class="n">last_line</span> <span class="o">=</span> <span class="vi">@scanner</span><span class="o">.</span><span class="n">rest</span><span class="o">.</span><span class="n">to_s</span><span class="p">)</span><span class="o">.</span><span class="n">size</span> <span class="o">></span> <span class="mi">0</span>
<span class="n">scan_line</span><span class="p">(</span><span class="n">last_line</span><span class="p">)</span>
<span class="k">end</span></pre></div>
</td>
</tr>
<tr id='section-13'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-13">¶</a>
</div>
<p>Finish up the final fragment. Finishing a fragment will detect any
attributes (hidden, signature, reply), and join each line into a
string.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="n">finish_fragment</span>
<span class="vi">@scanner</span> <span class="o">=</span> <span class="vi">@fragment</span> <span class="o">=</span> <span class="kp">nil</span></pre></div>
</td>
</tr>
<tr id='section-14'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-14">¶</a>
</div>
<p>Now that parsing is done, reverse the order.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="vi">@fragments</span><span class="o">.</span><span class="n">reverse!</span>
<span class="nb">self</span>
<span class="k">end</span>
<span class="kp">private</span>
<span class="no">EMPTY</span> <span class="o">=</span> <span class="s2">""</span><span class="o">.</span><span class="n">freeze</span></pre></div>
</td>
</tr>
<tr id='section-Line-by-Line_Parsing'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-Line-by-Line_Parsing">¶</a>
</div>
<h2>Line-by-Line Parsing</h2>
</td>
<td class=code>
<div class='highlight'><pre></pre></div>
</td>
</tr>
<tr id='section-16'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-16">¶</a>
</div>
<p>Scans the given line of text and figures out which fragment it belongs
to.</p>
<p>line – A String line of text from the email.</p>
<p>Returns nothing.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="k">def</span> <span class="nf">scan_line</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
<span class="n">line</span><span class="o">.</span><span class="n">chomp!</span><span class="p">(</span><span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">)</span>
<span class="n">line</span><span class="o">.</span><span class="n">lstrip!</span></pre></div>
</td>
</tr>
<tr id='section-17'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-17">¶</a>
</div>
<p>We’re looking for leading <code>></code>’s to see if this line is part of a
quoted Fragment.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="n">line_levels</span> <span class="o">=</span> <span class="n">line</span> <span class="o">=~</span> <span class="sr">/(>+)$/</span> <span class="p">?</span> <span class="vg">$1</span><span class="o">.</span><span class="n">size</span> <span class="p">:</span> <span class="mi">0</span></pre></div>
</td>
</tr>
<tr id='section-18'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-18">¶</a>
</div>
<p>Mark the current Fragment as a signature if the current line is empty
and the Fragment starts with a common signature indicator.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="k">if</span> <span class="vi">@fragment</span> <span class="o">&&</span> <span class="n">line</span> <span class="o">==</span> <span class="no">EMPTY</span>
<span class="k">if</span> <span class="vi">@fragment</span><span class="o">.</span><span class="n">lines</span><span class="o">.</span><span class="n">last</span> <span class="o">=~</span> <span class="sr">/[\-\_]$/</span>
<span class="vi">@fragment</span><span class="o">.</span><span class="n">signature</span> <span class="o">=</span> <span class="kp">true</span>
<span class="n">finish_fragment</span>
<span class="k">end</span>
<span class="k">end</span></pre></div>
</td>
</tr>
<tr id='section-19'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-19">¶</a>
</div>
<p>If the line matches the current fragment, add it. Note that a common
reply header also counts as part of the quoted Fragment, even though
it doesn’t start with <code>></code>.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="k">if</span> <span class="vi">@fragment</span> <span class="o">&&</span>
<span class="p">((</span><span class="vi">@fragment</span><span class="o">.</span><span class="n">quoted?</span> <span class="o">!=</span> <span class="n">line_levels</span><span class="o">.</span><span class="n">zero?</span><span class="p">)</span> <span class="o">||</span>
<span class="p">(</span><span class="vi">@fragment</span><span class="o">.</span><span class="n">quoted?</span> <span class="o">&&</span> <span class="n">quote_header?</span><span class="p">(</span><span class="n">line</span><span class="p">)))</span>
<span class="vi">@fragment</span><span class="o">.</span><span class="n">lines</span> <span class="o"><<</span> <span class="n">line</span></pre></div>
</td>
</tr>
<tr id='section-20'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-20">¶</a>
</div>
<p>Otherwise, finish the fragment and start a new one.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="k">else</span>
<span class="n">finish_fragment</span>
<span class="vi">@fragment</span> <span class="o">=</span> <span class="no">Fragment</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="o">!</span><span class="n">line_levels</span><span class="o">.</span><span class="n">zero?</span><span class="p">,</span> <span class="n">line</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span></pre></div>
</td>
</tr>
<tr id='section-21'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-21">¶</a>
</div>
<p>Detects if a given line is a header above a quoted area. It is only
checked for lines preceding quoted regions.</p>
<p>line – A String line of text from the email.</p>
<p>Returns true if the line is a valid header, or false.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="k">def</span> <span class="nf">quote_header?</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
<span class="n">line</span> <span class="o">=~</span> <span class="sr">/^:etorw.*nO$/</span>
<span class="k">end</span></pre></div>
</td>
</tr>
<tr id='section-22'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-22">¶</a>
</div>
<p>Builds the fragment string and reverses it, after all lines have been
added. It also checks to see if this Fragment is hidden. The hidden
Fragment check reads from the bottom to the top.</p>
<p>Any quoted Fragments or signature Fragments are marked hidden if they
are below any visible Fragments. Visible Fragments are expected to
contain original content by the author. If they are below a quoted
Fragment, then the Fragment should be visible to give context to the
reply.</p>
<pre><code>some original text (visible)
> do you have any two's? (quoted, visible)
Go fish! (visible)
> --
> Player 1 (quoted, hidden)
--
Player 2 (signature, hidden)
</code></pre>
</td>
<td class=code>
<div class='highlight'><pre> <span class="k">def</span> <span class="nf">finish_fragment</span>
<span class="k">if</span> <span class="vi">@fragment</span>
<span class="vi">@fragment</span><span class="o">.</span><span class="n">finish</span>
<span class="k">if</span> <span class="o">!</span><span class="vi">@found_visible</span>
<span class="k">if</span> <span class="vi">@fragment</span><span class="o">.</span><span class="n">quoted?</span> <span class="o">||</span> <span class="vi">@fragment</span><span class="o">.</span><span class="n">signature?</span> <span class="o">||</span>
<span class="vi">@fragment</span><span class="o">.</span><span class="n">to_s</span><span class="o">.</span><span class="n">strip</span> <span class="o">==</span> <span class="no">EMPTY</span>
<span class="vi">@fragment</span><span class="o">.</span><span class="n">hidden</span> <span class="o">=</span> <span class="kp">true</span>
<span class="k">else</span>
<span class="vi">@found_visible</span> <span class="o">=</span> <span class="kp">true</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="vi">@fragments</span> <span class="o"><<</span> <span class="vi">@fragment</span>
<span class="k">end</span>
<span class="vi">@fragment</span> <span class="o">=</span> <span class="kp">nil</span>
<span class="k">end</span>
<span class="k">end</span></pre></div>
</td>
</tr>
<tr id='section-Fragments'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-Fragments">¶</a>
</div>
<h2>Fragments</h2>
</td>
<td class=code>
<div class='highlight'><pre></pre></div>
</td>
</tr>
<tr id='section-24'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-24">¶</a>
</div>
<p>Represents a group of paragraphs in the email sharing common attributes.
Paragraphs should get their own fragment if they are a quoted area or a
signature.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="k">class</span> <span class="nc">Fragment</span> <span class="o"><</span> <span class="no">Struct</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="ss">:quoted</span><span class="p">,</span> <span class="ss">:signature</span><span class="p">,</span> <span class="ss">:hidden</span><span class="p">)</span></pre></div>
</td>
</tr>
<tr id='section-25'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-25">¶</a>
</div>
<p>This is an Array of String lines of content. Since the content is
reversed, this array is backwards, and contains reversed strings.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="kp">attr_reader</span> <span class="ss">:lines</span><span class="p">,</span></pre></div>
</td>
</tr>
<tr id='section-26'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-26">¶</a>
</div>
<p>This is reserved for the joined String that is build when this Fragment
is finished.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="ss">:content</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">quoted</span><span class="p">,</span> <span class="n">first_line</span><span class="p">)</span>
<span class="nb">self</span><span class="o">.</span><span class="n">signature</span> <span class="o">=</span> <span class="nb">self</span><span class="o">.</span><span class="n">hidden</span> <span class="o">=</span> <span class="kp">false</span>
<span class="nb">self</span><span class="o">.</span><span class="n">quoted</span> <span class="o">=</span> <span class="n">quoted</span>
<span class="vi">@lines</span> <span class="o">=</span> <span class="o">[</span><span class="n">first_line</span><span class="o">]</span>
<span class="vi">@content</span> <span class="o">=</span> <span class="kp">nil</span>
<span class="vi">@lines</span><span class="o">.</span><span class="n">compact!</span>
<span class="k">end</span>
<span class="k">alias</span> <span class="n">quoted?</span> <span class="n">quoted</span>
<span class="k">alias</span> <span class="n">signature?</span> <span class="n">signature</span>
<span class="k">alias</span> <span class="n">hidden?</span> <span class="n">hidden</span></pre></div>
</td>
</tr>
<tr id='section-27'>
<td class=docs>
<div class="pilwrap">
<a class="pilcrow" href="#section-27">¶</a>
</div>
<p>Builds the string content by joining the lines and reversing them.</p>
<p>Returns nothing.</p>
</td>
<td class=code>
<div class='highlight'><pre> <span class="k">def</span> <span class="nf">finish</span>
<span class="vi">@content</span> <span class="o">=</span> <span class="vi">@lines</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">)</span>
<span class="vi">@lines</span> <span class="o">=</span> <span class="kp">nil</span>
<span class="vi">@content</span><span class="o">.</span><span class="n">reverse!</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">to_s</span>
<span class="vi">@content</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">inspect</span>
<span class="nb">to_s</span><span class="o">.</span><span class="n">inspect</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></pre></div>
</td>
</tr>
</table>
</div>
</body>
|
