Planet Docunext

Apache FastCGI

2010-03-08T23:46:58+00:00

← Older revision		Revision as of 23:46, 8 March 2010
Line 1:		Line 1:
	<html>		<html>
-	<img src="/s/img/2009/q2/Helena_Mattsson.jpg" style="display: inline; float: right; padding-left: 20px; padding-right: 20px; margin-top: -15px;"/>	+	<img src="http://www-01.evenserver.com/s/img/2009/q2/Helena_Mattsson.jpg" style="display: inline; float: right; padding-left: 20px; padding-right: 20px; margin-top: -15px;"/>
	</html>		</html>
	== Summary ==		== Summary ==

Apache FCGI

2010-03-08T23:46:09+00:00

Redirected page to Apache FastCGI

New page

#REDIRECT [[Apache FastCGI]]

Ruby Gem: ruby-aaws

2010-03-08T22:53:38+00:00

This is an awesome gem, though I ran into many errors when I first started to try it out. Here's the code I was trying:

require 'amazon/aws'
require 'amazon/aws/search'
require 'pp'

include Amazon::AWS
include Amazon::AWS::Search

ASSOCIATES_ID = "removed"
KEY_ID = "removed"

#req = Request.new(KEY_ID, ASSOCIATES_ID, 'us', false)

is = ItemSearch.new( 'Books', { 'Title' => 'Ruby' } )
is.response_group = ResponseGroup.new( :Small )
req = Request.new(KEY_ID, ASSOCIATES_ID, 'us', false)
resp = req.search( is )

pp(resp.item_search_response)

pp(resp)

I first had trouble figuring out that I needed to put my AWS "secret" key in ~/.amazonrc.

I was getting UTF-8 errors, too. I think adding this to my ~/.amazonrc fixed the problem:

encoding = 'UTF-8'

Hmmm. I guess using the cache also causes UTF-8 related errors. UPDATE: Finally, late last night I figured it out. The [example aaws code](http://www.somelifeblog.com/2008/12/ruby-amazon-associates-web-services-aws.html) I was basing my tests on uses ItemLookup, which has an item_lookup_response object. When I switched to searching, item_lookup_response is not there. I eventually switched to item_search_response, but had cleared out the rest of the example for debugging. Adding it back in returned the right set.

is = ItemSearch.new( 'Books', { 'Title' => 'Ruby' } )
is.response_group = ResponseGroup.new( :Small ) 
req = Request.new(KEY_ID, ASSOCIATES_ID, 'us', true)
resp = req.search( is )

item_sets = resp.item_search_response[0].items
  item_sets[0..3].each do |item_set|
  item_set.item.each do |item|
    attribs = item.item_attributes[0]
    puts attribs
  end
end

The way the secret key is handled seems odd. Maybe there is a better way? The ruby-aaws RDoc says its the only way though.

UPDATE: There are different ways to set where the amazonrc file is. In my case, running unicorn via daemontools caused ENV['HOME'] to be nil, resulting in amazon.rb trying to evaluate ENV['HOMEDRIVE'] + ENV['HOMEPATH'], causing a NoMethodError that was very difficult to decipher (running unicorn with the "-d" option after invoking the Ruby interpreter helped track down the cause).

I had tried setting $AMAZONRCFILE at first, then $AMAZONRCDIR, but that didn't work I noticed in amazon.rb that it was looking for ENV['AMAZONRCDIR'], so I set it in config.ru and voila - functionality!!

UPDATE:

I've send Ian an email:

Hi Ian,

Nice work on the ruby-aaws gem. I'm using it with Sinatra and Unicorn, which is run by daemontools. For whatever reason, ENV['HOME'] was resulting in nil, causing it to try ENV['HOMEDRIVE'] + ENV['HOMEPATH'], which for me resulted in a NoMethodError.

Setting ENV['AMAZONRCDIR'] fixed the issue for me, but perhaps you'd consider putting an exception trap there, as the NoMethodError is so vague it was tough to track down.

Thanks for this sweet gem!

- Albert

PS - I've blogged about the gem, too:

http://www.ruby.code-experiments.com/blog/2010/03/ruby-gem-ruby-aaws.html

Still not sure what the problem is with UTF-8 and caching....

--
http://www.docunext.com/

Bricolage

2010-03-08T15:56:50+00:00

← Older revision		Revision as of 15:56, 8 March 2010
Line 70:		Line 70:
	== Related Pages ==		== Related Pages ==
	* [[Established Sites]]		* [[Established Sites]]
		+	* [[Catalyst]]
		+	* [[Dancer]]

	== External Sites ==		== External Sites ==
-	http://www.bricolage.cc/	+	* http://www.bricolage.cc/


-	[[Category:CMS]]	+	[[Category: CMS]]
-	[[Category:Perl]]	+	[[Category: Perl]]

Surf

2010-03-08T07:05:25+00:00

Created page with 'Surf is a web browser based on webkit that is very little other than a web browser. I like it! == External Links == * http://surf.suckless.org/ Category: Web Browsers'

New page

Surf is a web browser based on webkit that is very little other than a web browser. I like it!

== External Links ==
* http://surf.suckless.org/

[[Category: Web Browsers]]

Compiz

2010-03-08T06:38:09+00:00

← Older revision		Revision as of 06:38, 8 March 2010
Line 1:		Line 1:
-	Compiz is a window manager for [[X11]].	+	Compiz is a window manager for [[X11]]. A similar project named Beryl meged with Compiz a few years back.

	It is full of eye-candy.		It is full of eye-candy.

Ruby Pony

2010-03-08T05:57:02+00:00

← Older revision		Revision as of 05:57, 8 March 2010
(One intermediate revision not shown)
Line 11:		Line 11:
	:auth => :plain,		:auth => :plain,
	:domain => "example.com"		:domain => "example.com"
-	}	+	},
		+	:from => 'me@example.com', :subject => 'hi', :body => 'Hello there.')
		+	</pre>
		+
		+
		+	== Ruby Pony Errors ==
		+	<pre class="sh_log">
		+	unicorn worker[0] --env development -l 3000: symbol lookup error: /var/lib/gems/1.9.1/gems/tmail-1.2.7.1/ext/tmailscanner/tmail/tmailscanner.so: undefined symbol: rb_get_kcode
	</pre>		</pre>

Pony = Easy Emailing with Rubby

2010-03-08T05:55:18+00:00

I just found out about Pony, a Ruby gem for sending emails.

The cool thing about Pony is how simple it is. Good work, people!

  Pony.mail(:to => 'email@example.com', :via => :smtp, :smtp => {
    :host     => 'smtp.example.com',
    :port     => '25',
    :user     => 'username',
    :password => 'passkey',
    :auth     => :plain,
    :domain   => "example.com"
  },
  :from => 'me@example.com', :subject => 'hi', :body => 'Hello there.')

I'm planning on using this with Sinatra, but I'll probably end up using it with some Ruby scripts as well.

Speaking of which, I bought a book about using Ruby for systems administration awhile back. It had some really cool ideas in it, but I wasn't as comfortable with Ruby back then as I am now.

I definitely have to give this book a review!

More information: Ruby Pony @ Docunext

Compiz

2010-03-08T04:02:30+00:00

← Older revision		Revision as of 04:02, 8 March 2010
Line 1:		Line 1:
	Compiz is a window manager for [[X11]].		Compiz is a window manager for [[X11]].
		+
		+	It is full of eye-candy.
		+
		+
		+	[[Category: X11]]

docunext pushed to master at docunext/regdel

2010-03-08T02:42:28+00:00

docunext committed ce3ba21

new
docunext committed 6742154

new

docunext pushed to master at docunext/chimailmadmin

2010-03-08T02:38:47+00:00

docunext committed 6df6498

new
docunext committed f7e2a75

new

docunext pushed to master at docunext/1bb02b59

2010-03-08T02:25:12+00:00

docunext committed b0725a8

use locally

User Interface Design

2010-03-07T20:49:08+00:00

Redirected page to Graphical User Interface

New page

#REDIRECT [[Graphical User Interface]]

STDIN

2010-03-07T20:24:23+00:00

← Older revision		Revision as of 20:24, 7 March 2010
Line 1:		Line 1:
	Standard Input		Standard Input
		+
		+
		+	[[Category: Command Line]]

Sinatra and Rack Sessions

2010-03-07T02:56:56+00:00

I think that it took me awhile to figure out that in order to use memcache for sessions, "require 'memcache'" must come before "require 'rack'".

I say I think because it works now, but I'm not 100% certain that moving the memcache require statement was the resolution.

In other news, this looks cool:

git://github.com/JunKikuchi/rack-session-stack.git

Redis

2010-03-06T23:27:48+00:00

← Older revision		Revision as of 23:27, 6 March 2010
Line 12:		Line 12:

	Sweet!		Sweet!
		+
		+	One thing I wanted to do with a Redis list that I don't think is natively possible is to get the natural sort index position of a key. It is possible to get keys by index number, but apparently not vice-versa / the other way around. I think its possible if you get fancy with the key names, but since I was just dealing with a basic list, this is what I did (in [[Ruby]]):
		+
		+	<pre class="sh_ruby">
		+	pkgpos = @rdis.list_range('pkgs',0,-1).index(@pkg)
		+	</pre>
		+
		+	Basically what I'm doing is getting the entire list as an array, and then using [[Ruby]]'s capability to do what I needed.

	== Redis Configuration ==		== Redis Configuration ==

Main Page

2010-03-06T23:10:43+00:00

← Older revision		Revision as of 23:10, 6 March 2010
Line 11:		Line 11:

	We have the [http://www.docunext.com/blog/ Docunext Tech Stuff Blog] of course, this wiki (which is just recently back from hiding), and have plans to add a bunch more stuff, including some file repositories and development tools.		We have the [http://www.docunext.com/blog/ Docunext Tech Stuff Blog] of course, this wiki (which is just recently back from hiding), and have plans to add a bunch more stuff, including some file repositories and development tools.
-
-	~~We also just launched the [[Docunext Forums]].~~

	=== [[Docunext Wiki]] ===		=== [[Docunext Wiki]] ===

docunext pushed to master at docunext/chimailmadmin

2010-03-06T21:34:59+00:00

docunext committed 0840977

spamassassin edit form

Mail::Spamassassin::Conf

2010-03-06T21:13:28+00:00

Created page with '<pre> Mail::SpamAssassin::Conf(3pm) User Contributed Perl Documentation Mail::SpamAssassin::Conf(3pm...'

New page

<pre>
Mail::SpamAssassin::Conf(3pm) User Contributed Perl Documentation Mail::SpamAssassin::Conf(3pm)

NAME
Mail::SpamAssassin::Conf - SpamAssassin configuration file

SYNOPSIS
# a comment

rewrite_header Subject *****SPAM*****

full PARA_A_2_C_OF_1618 /Paragraph .a.{0,10}2.{0,10}C. of S. 1618/i
describe PARA_A_2_C_OF_1618 Claims compliance with senate bill 1618

header FROM_HAS_MIXED_NUMS From =~ /\d+[a-z]+\d+\S*@/i
describe FROM_HAS_MIXED_NUMS From: contains numbers mixed in with letters

score A_HREF_TO_REMOVE 2.0

lang es describe FROM_FORGED_HOTMAIL Forzado From: simula ser de hotmail.com

lang pt_BR report O programa detetor de Spam ZOE [...]

DESCRIPTION
SpamAssassin is configured using traditional UNIX-style configuration files, loaded from the "/usr/share/spamassassin" and "/etc/spamassassin" directories.

The following web page lists the most important configuration settings used to configure SpamAssassin; novices are encouraged to read it first:

http://wiki.apache.org/spamassassin/ImportantInitialConfigItems

FILE FORMAT
The "#" character starts a comment, which continues until end of line. NOTE: if the "#" character is to be used as part of a rule or configuration option, it must be
escaped with a backslash. i.e.: "\#"

Whitespace in the files is not significant, but please note that starting a line with whitespace is deprecated, as we reserve its use for multi-line rule definitions,
at some point in the future.

Currently, each rule or configuration setting must fit on one-line; multi-line settings are not supported yet.

File and directory paths can use "~" to refer to the user's home directory, but no other shell-style path extensions such as globing or "~user/" are supported.

Where appropriate below, default values are listed in parentheses.

USER PREFERENCES
The following options can be used in both site-wide ("local.cf") and user-specific ("user_prefs") configuration files to customize how SpamAssassin handles incoming
email messages.

SCORING OPTIONS
required_score n.nn (default: 5)
Set the score required before a mail is considered spam. "n.nn" can be an integer or a real number. 5.0 is the default setting, and is quite aggressive; it
would be suitable for a single-user setup, but if you're an ISP installing SpamAssassin, you should probably set the default to be more conservative, like 8.0 or
10.0. It is not recommended to automatically delete or discard messages marked as spam, as your users will complain, but if you choose to do so, only delete
messages with an exceptionally high score such as 15.0 or higher. This option was previously known as "required_hits" and that name is still accepted, but is
deprecated.

score SYMBOLIC_TEST_NAME n.nn [ n.nn n.nn n.nn ]
Assign scores (the number of points for a hit) to a given test. Scores can be positive or negative real numbers or integers. "SYMBOLIC_TEST_NAME" is the
symbolic name used by SpamAssassin for that test; for example, 'FROM_ENDS_IN_NUMS'.

If only one valid score is listed, then that score is always used for a test.

If four valid scores are listed, then the score that is used depends on how SpamAssassin is being used. The first score is used when both Bayes and network tests
are disabled (score set 0). The second score is used when Bayes is disabled, but network tests are enabled (score set 1). The third score is used when Bayes is
enabled and network tests are disabled (score set 2). The fourth score is used when Bayes is enabled and network tests are enabled (score set 3).

Setting a rule's score to 0 will disable that rule from running.

If any of the score values are surrounded by parenthesis '()', then all of the scores in the line are considered to be relative to the already set score. ie:
'(3)' means increase the score for this rule by 3 points in all score sets. '(3) (0) (3) (0)' means increase the score for this rule by 3 in score sets 0 and 2
only.

If no score is given for a test by the end of the configuration, a default score is assigned: a score of 1.0 is used for all tests, except those who names begin
with 'T_' (this is used to indicate a rule in testing) which receive 0.01.

Note that test names which begin with '__' are indirect rules used to compose meta-match rules and can also act as prerequisites to other rules. They are not
scored or listed in the 'tests hit' reports, but assigning a score of 0 to an indirect rule will disable it from running.

WHITELIST AND BLACKLIST OPTIONS
whitelist_from add@ress.com
Used to whitelist sender addresses which send mail that is often tagged (incorrectly) as spam.

Use of this setting is not recommended, since it blindly trusts the message, which is routinely and easily forged by spammers and phish senders. The recommended
solution is to instead use "whitelist_auth" or other authenticated whitelisting methods, or "whitelist_from_rcvd".

Whitelist and blacklist addresses are now file-glob-style patterns, so "friend@somewhere.com", "*@isp.com", or "*.domain.net" will all work. Specifically, "*"
and "?" are allowed, but all other metacharacters are not. Regular expressions are not used for security reasons.

Multiple addresses per line, separated by spaces, is OK. Multiple "whitelist_from" lines is also OK.

The headers checked for whitelist addresses are as follows: if "Resent-From" is set, use that; otherwise check all addresses taken from the following set of
headers:

Envelope-Sender
Resent-Sender
X-Envelope-From
From

In addition, the "envelope sender" data, taken from the SMTP envelope data where this is available, is looked up. See "envelope_sender_header".

e.g.

whitelist_from joe@example.com fred@example.com
whitelist_from *@example.com

unwhitelist_from add@ress.com
Used to override a default whitelist_from entry, so for example a distribution whitelist_from can be overridden in a local.cf file, or an individual user can
override a whitelist_from entry in their own "user_prefs" file. The specified email address has to match exactly the address previously used in a whitelist_from
line.

e.g.

unwhitelist_from joe@example.com fred@example.com
unwhitelist_from *@example.com

whitelist_from_rcvd addr@lists.sourceforge.net sourceforge.net
Use this to supplement the whitelist_from addresses with a check against the Received headers. The first parameter is the address to whitelist, and the second is
a string to match the relay's rDNS.

This string is matched against the reverse DNS lookup used during the handover from the internet to your internal network's mail exchangers. It can either be the
full hostname, or the domain component of that hostname. In other words, if the host that connected to your MX had an IP address that mapped to
'sendinghost.spamassassin.org', you should specify "sendinghost.spamassassin.org" or just "spamassassin.org" here.

Note that this requires that "internal_networks" be correct. For simple cases, it will be, but for a complex network you may get better results by setting that
parameter.

It also requires that your mail exchangers be configured to perform DNS reverse lookups on the connecting host's IP address, and to record the result in the
generated Received: header.

e.g.

whitelist_from_rcvd joe@example.com example.com
whitelist_from_rcvd *@axkit.org sergeant.org

def_whitelist_from_rcvd addr@lists.sourceforge.net sourceforge.net
Same as "whitelist_from_rcvd", but used for the default whitelist entries in the SpamAssassin distribution. The whitelist score is lower, because these are often
targets for spammer spoofing.

whitelist_allows_relays add@ress.com
Specify addresses which are in "whitelist_from_rcvd" that sometimes send through a mail relay other than the listed ones. By default mail with a From address that
is in "whitelist_from_rcvd" that does not match the relay will trigger a forgery rule. Including the address in "whitelist_allows_relay" prevents that.

Whitelist and blacklist addresses are now file-glob-style patterns, so "friend@somewhere.com", "*@isp.com", or "*.domain.net" will all work. Specifically, "*"
and "?" are allowed, but all other metacharacters are not. Regular expressions are not used for security reasons.

Multiple addresses per line, separated by spaces, is OK. Multiple "whitelist_allows_relays" lines is also OK.

The specified email address does not have to match exactly the address previously used in a whitelist_from_rcvd line as it is compared to the address in the
header.

e.g.

whitelist_allows_relays joe@example.com fred@example.com
whitelist_allows_relays *@example.com
unwhitelist_from_rcvd add@ress.com
Used to override a default whitelist_from_rcvd entry, so for example a distribution whitelist_from_rcvd can be overridden in a local.cf file, or an individual
user can override a whitelist_from_rcvd entry in their own "user_prefs" file.

The specified email address has to match exactly the address previously used in a whitelist_from_rcvd line.

e.g.

unwhitelist_from_rcvd joe@example.com fred@example.com
unwhitelist_from_rcvd *@axkit.org

blacklist_from add@ress.com
Used to specify addresses which send mail that is often tagged (incorrectly) as non-spam, but which the user doesn't want. Same format as "whitelist_from".

unblacklist_from add@ress.com
Used to override a default blacklist_from entry, so for example a distribution blacklist_from can be overridden in a local.cf file, or an individual user can
override a blacklist_from entry in their own "user_prefs" file. The specified email address has to match exactly the address previously used in a blacklist_from
line.

e.g.

unblacklist_from joe@example.com fred@example.com
unblacklist_from *@spammer.com

whitelist_to add@ress.com
If the given address appears as a recipient in the message headers (Resent-To, To, Cc, obvious envelope recipient, etc.) the mail will be whitelisted. Useful if
you're deploying SpamAssassin system-wide, and don't want some users to have their mail filtered. Same format as "whitelist_from".

There are three levels of To-whitelisting, "whitelist_to", "more_spam_to" and "all_spam_to". Users in the first level may still get some spammish mails blocked,
but users in "all_spam_to" should never get mail blocked.

The headers checked for whitelist addresses are as follows: if "Resent-To" or "Resent-Cc" are set, use those; otherwise check all addresses taken from the
following set of headers:

To
Cc
Apparently-To
Delivered-To
Envelope-Recipients
Apparently-Resent-To
X-Envelope-To
Envelope-To
X-Delivered-To
X-Original-To
X-Rcpt-To
X-Real-To

more_spam_to add@ress.com
See above.

all_spam_to add@ress.com
See above.

blacklist_to add@ress.com
If the given address appears as a recipient in the message headers (Resent-To, To, Cc, obvious envelope recipient, etc.) the mail will be blacklisted. Same
format as "blacklist_from".

whitelist_auth add@ress.com
Used to specify addresses which send mail that is often tagged (incorrectly) as spam. This is different from "whitelist_from" and "whitelist_from_rcvd" in that
it first verifies that the message was sent by an authorized sender for the address, before whitelisting.

Authorization is performed using one of the installed sender-authorization schemes: SPF (using "Mail::SpamAssassin::Plugins::SPF"), Domain Keys (using
"Mail::SpamAssassin::Plugins::DomainKeys"), or DKIM (using "Mail::SpamAssassin::Plugins::DKIM"). Note that those plugins must be active, and working, for this to
operate.

Using "whitelist_auth" is roughly equivalent to specifying duplicate "whitelist_from_spf", "whitelist_from_dk", and "whitelist_from_dkim" lines for each of the
addresses specified.

e.g.

whitelist_auth joe@example.com fred@example.com
whitelist_auth *@example.com

def_whitelist_auth add@ress.com
Same as "whitelist_auth", but used for the default whitelist entries in the SpamAssassin distribution. The whitelist score is lower, because these are often
targets for spammer spoofing.

unwhitelist_auth add@ress.com
Used to override a "whitelist_auth" entry. The specified email address has to match exactly the address previously used in a "whitelist_auth" line.

e.g.

unwhitelist_auth joe@example.com fred@example.com
unwhitelist_auth *@example.com

BASIC MESSAGE TAGGING OPTIONS
rewrite_header { subject | from | to } STRING
By default, suspected spam messages will not have the "Subject", "From" or "To" lines tagged to indicate spam. By setting this option, the header will be tagged
with "STRING" to indicate that a message is spam. For the From or To headers, this will take the form of an RFC 2822 comment following the address in parantheses.
For the Subject header, this will be prepended to the original subject. Note that you should only use the _REQD_ and _SCORE_ tags when rewriting the Subject
header if "report_safe" is 0. Otherwise, you may not be able to remove the SpamAssassin markup via the normal methods. More information about tags is explained
below in the TEMPLATE TAGS section.

Parentheses are not permitted in STRING if rewriting the From or To headers. (They will be converted to square brackets.)

If "rewrite_header subject" is used, but the message being rewritten does not already contain a "Subject" header, one will be created.

A null value for "STRING" will remove any existing rewrite for the specified header.

add_header { spam | ham | all } header_name string
Customized headers can be added to the specified type of messages (spam, ham, or "all" to add to either). All headers begin with "X-Spam-" (so a "header_name"
Foo will generate a header called X-Spam-Foo). header_name is restricted to the character set [A-Za-z0-9_-].

"string" can contain tags as explained below in the TEMPLATE TAGS section. You can also use "\n" and "\t" in the header to add newlines and tabulators as
desired. A backslash has to be written as \\, any other escaped chars will be silently removed.

All headers will be folded if fold_headers is set to 1. Note: Manually adding newlines via "\n" disables any further automatic wrapping (ie: long header lines are
possible). The lines will still be properly folded (marked as continuing) though.

You can customize existing headers with add_header (only the specified subset of messages will be changed).

See also "clear_headers" for removing headers.

Here are some examples (these are the defaults, note that Checker-Version can not be changed or removed):

add_header spam Flag _YESNOCAPS_
add_header all Status _YESNO_, score=_SCORE_ required=_REQD_ tests=_TESTS_ autolearn=_AUTOLEARN_ version=_VERSION_
add_header all Level _STARS(*)_
add_header all Checker-Version SpamAssassin _VERSION_ (_SUBVERSION_) on _HOSTNAME_

remove_header { spam | ham | all } header_name
Headers can be removed from the specified type of messages (spam, ham, or "all" to remove from either). All headers begin with "X-Spam-" (so "header_name" will
be appended to "X-Spam-").

See also "clear_headers" for removing all the headers at once.

Note that X-Spam-Checker-Version is not removable because the version information is needed by mail administrators and developers to debug problems. Without at
least one header, it might not even be possible to determine that SpamAssassin is running.

clear_headers
Clear the list of headers to be added to messages. You may use this before any add_header options to prevent the default headers from being added to the message.

Note that X-Spam-Checker-Version is not removable because the version information is needed by mail administrators and developers to debug problems. Without at
least one header, it might not even be possible to determine that SpamAssassin is running.

report_safe ( 0 | 1 | 2 ) (default: 1)
if this option is set to 1, if an incoming message is tagged as spam, instead of modifying the original message, SpamAssassin will create a new report message and
attach the original message as a message/rfc822 MIME part (ensuring the original message is completely preserved, not easily opened, and easier to recover).

If this option is set to 2, then original messages will be attached with a content type of text/plain instead of message/rfc822. This setting may be required for
safety reasons on certain broken mail clients that automatically load attachments without any action by the user. This setting may also make it somewhat more
difficult to extract or view the original message.

If this option is set to 0, incoming spam is only modified by adding some "X-Spam-" headers and no changes will be made to the body. In addition, a header named
X-Spam-Report will be added to spam. You can use the remove_header option to remove that header after setting report_safe to 0.

See report_safe_copy_headers if you want to copy headers from the original mail into tagged messages.

LANGUAGE OPTIONS
ok_locales xx [ yy zz ... ] (default: all)
This option is used to specify which locales are considered OK for incoming mail. Mail using the character sets that are allowed by this option will not be
marked as possibly being spam in a foreign language.

If you receive lots of spam in foreign languages, and never get any non-spam in these languages, this may help. Note that all ISO-8859-* character sets, and
Windows code page character sets, are always permitted by default.

Set this to "all" to allow all character sets. This is the default.

The rules "CHARSET_FARAWAY", "CHARSET_FARAWAY_BODY", and "CHARSET_FARAWAY_HEADERS" are triggered based on how this is set.

ok_locales all (allow all locales)
ok_locales en (only allow English)
ok_locales en ja zh (allow English, Japanese, and Chinese)

Note: if there are multiple ok_locales lines, only the last one is used.

Select the locales to allow from the list below:

en - Western character sets in general
ja - Japanese character sets
ko - Korean character sets
ru - Cyrillic character sets
th - Thai character sets
zh - Chinese (both simplified and traditional) character sets
normalize_charset ( 0 | 1) (default: 0)
Whether to detect character sets and normalize message content to Unicode. Requires the Encode::Detect module, HTML::Parser version 3.46 or later, and Perl 5.8.5
or later.

NETWORK TEST OPTIONS
trusted_networks ip.add.re.ss[/mask] ... (default: none)
What networks or hosts are 'trusted' in your setup. Trusted in this case means that relay hosts on these networks are considered to not be potentially operated
by spammers, open relays, or open proxies. A trusted host could conceivably relay spam, but will not originate it, and will not forge header data. DNS blacklist
checks will never query for hosts on these networks.

See "http://wiki.apache.org/spamassassin/TrustPath" for more information.

MXes for your domain(s) and internal relays should also be specified using the "internal_networks" setting. When there are 'trusted' hosts that are not MXes or
internal relays for your domain(s) they should only be specified in "trusted_networks".

If a "/mask" is specified, it's considered a CIDR-style 'netmask', specified in bits. If it is not specified, but less than 4 octets are specified with a
trailing dot, that's considered a mask to allow all addresses in the remaining octets. If a mask is not specified, and there is not trailing dot, then just the
single IP address specified is used, as if the mask was "/32".

If a network or host address is prefaced by a "!" the network or host will be excluded (or included) in a first listed match fashion.

Note: 127/8 and ::1 are always included in trusted_networks, regardless of your config.

Examples:

trusted_networks 192.168/16 # all in 192.168.*.*
trusted_networks 212.17.35.15 # just that host
trusted_networks !10.0.1.5 10.0.1/24 # all in 10.0.1.* but not 10.0.1.5
trusted_networks DEAD:BEEF::/32 # all in that ipv6 prefix

This operates additively, so a "trusted_networks" line after another one will result in all those networks becoming trusted. To clear out the existing entries,
use "clear_trusted_networks".

If "trusted_networks" is not set and "internal_networks" is, the value of "internal_networks" will be used for this parameter.

If neither "trusted_networks" or "internal_networks" is set, a basic inference algorithm is applied. This works as follows:

� If the 'from' host has an IP address in a private (RFC 1918) network range, then it's trusted

� If there are authentication tokens in the received header, and the previous host was trusted, then this host is also trusted

� Otherwise this host, and all further hosts, are consider untrusted.

clear_trusted_networks
Empty the list of trusted networks.

internal_networks ip.add.re.ss[/mask] ... (default: none)
What networks or hosts are 'internal' in your setup. Internal means that relay hosts on these networks are considered to be MXes for your domain(s), or internal
relays. This uses the same format as "trusted_networks", above.

This value is used when checking 'dial-up' or dynamic IP address blocklists, in order to detect direct-to-MX spamming.

Trusted relays that accept mail directly from dial-up connections should not be listed in "internal_networks". List them only in "trusted_networks".

If "trusted_networks" is set and "internal_networks" is not, the value of "trusted_networks" will be used for this parameter.

If neither "trusted_networks" or "internal_networks" is set, no addresses will be considered local; in other words, any relays past the machine where SpamAssassin
is running will be considered external.

Every entry in "internal_networks" must appear in "trusted_networks"; in other words, "internal_networks" is always a subset of the trusted set.

Note: 127/8 and ::1 are always included in internal_networks, regardless of your config.

clear_internal_networks
Empty the list of internal networks.

msa_networks ip.add.re.ss[/mask] ... (default: none)
The networks or hosts are acting as MSAs in your setup. MSA means that the relay hosts on these networks accept mail from your own users and authenticates them
appropriately. These relays will never accept mail from hosts that aren't authenticated in some way. Examples of authentication include, IP lists, SMTP AUTH,
POP-before-SMTP, etc.

All relays found in the message headers after the MSA relay will take on the same trusted and internal classifcations as the MSA relay itself, as defined by your
trusted_networks and internal_networks configuration.

For example, if the MSA relay is trusted and internal so will all of the relays that precede it.

When using msa_networks to identify an MSA it is recommended that you treat that MSA as both trusted and internal. When an MSA is not included in msa_networks
you should treat the MSA as trusted but not internal, however if the MSA is also acting as an MX or intermediate relay you must always treat it as both trusted
and internal and ensure that the MSA includes visible auth tokens in its Received header to identify submission clients.

Warning: Never include an MSA that also acts as an MX (or is also an intermediate relay for an MX) or otherwise accepts mail from non-authenticated users in
msa_networks. Doing so will result in unknown external relays being trusted.

clear_msa_networks
Empty the list of msa networks.

always_trust_envelope_sender ( 0 | 1 ) (default: 0)
Trust the envelope sender even if the message has been passed through one or more trusted relays. See also "envelope_sender_header".

skip_rbl_checks ( 0 | 1 ) (default: 0)
By default, SpamAssassin will run RBL checks. If your ISP already does this for you, set this to 1.

dns_available { yes | test[: name1 name2...] | no } (default: test)
By default, SpamAssassin will query some default hosts on the internet to attempt to check if DNS is working or not. The problem is that it can introduce some
delay if your network connection is down, and in some cases it can wrongly guess that DNS is unavailable because the test connections failed. SpamAssassin
includes a default set of 13 servers, among which 3 are picked randomly.

You can however specify your own list by specifying

dns_available test: domain1.tld domain2.tld domain3.tld

Please note, the DNS test queries for NS records.

SpamAssassin's network rules are run in parallel. This can cause overhead in terms of the number of file descriptors required; it is recommended that the minimum
limit on file descriptors be raised to at least 256 for safety.

dns_test_interval n (default: 600 seconds)
If dns_available is set to 'test' (which is the default), the dns_test_interval time in number of seconds will tell SpamAssassin how often to retest for working
DNS.

dns_options rotate (default: empty)
If set to 'rotate', this causes SpamAssassin to choose a DNS server at random from all servers listed in "/etc/resolv.conf" every 'dns_test_interval' seconds,
effectively spreading the load over all currently available DNS servers when there are many spamd workers.

LEARNING OPTIONS
use_bayes ( 0 | 1 ) (default: 1)
Whether to use the naive-Bayesian-style classifier built into SpamAssassin. This is a master on/off switch for all Bayes-related operations.

use_bayes_rules ( 0 | 1 ) (default: 1)
Whether to use rules using the naive-Bayesian-style classifier built into SpamAssassin. This allows you to disable the rules while leaving auto and manual
learning enabled.

bayes_auto_learn ( 0 | 1 ) (default: 1)
Whether SpamAssassin should automatically feed high-scoring mails (or low-scoring mails, for non-spam) into its learning systems. The only learning system
supported currently is a naive-Bayesian-style classifier.

See the documentation for the "Mail::SpamAssassin::Plugin::AutoLearnThreshold" plugin module for details on how Bayes auto-learning is implemented by default.

bayes_ignore_header header_name
If you receive mail filtered by upstream mail systems, like a spam-filtering ISP or mailing list, and that service adds new headers (as most of them do), these
headers may provide inappropriate cues to the Bayesian classifier, allowing it to take a "short cut". To avoid this, list the headers using this setting.
Example:

bayes_ignore_header X-Upstream-Spamfilter
bayes_ignore_header X-Upstream-SomethingElse

bayes_ignore_from add@ress.com
Bayesian classification and autolearning will not be performed on mail from the listed addresses. Program "sa-learn" will also ignore the listed addresses if it
is invoked using the "--use-ignores" option. One or more addresses can be listed, see "whitelist_from".

Spam messages from certain senders may contain many words that frequently occur in ham. For example, one might read messages from a preferred bookstore but also
get unwanted spam messages from other bookstores. If the unwanted messages are learned as spam then any messages discussing books, including the preferred
bookstore and antiquarian messages would be in danger of being marked as spam. The addresses of the annoying bookstores would be listed. (Assuming they were
halfway legitimate and didn't send you mail through myriad affiliates.)

Those who have pieces of spam in legitimate messages or otherwise receive ham messages containing potentially spammy words might fear that some spam messages
might be in danger of being marked as ham. The addresses of the spam mailing lists, correspondents, etc. would be listed.

bayes_ignore_to add@ress.com
Bayesian classification and autolearning will not be performed on mail to the listed addresses. See "bayes_ignore_from" for details.

bayes_min_ham_num (Default: 200)
bayes_min_spam_num (Default: 200)
To be accurate, the Bayes system does not activate until a certain number of ham (non-spam) and spam have been learned. The default is 200 of each ham and spam,
but you can tune these up or down with these two settings.

bayes_learn_during_report (Default: 1)
The Bayes system will, by default, learn any reported messages ("spamassassin -r") as spam. If you do not want this to happen, set this option to 0.

bayes_sql_override_username
Used by BayesStore::SQL storage implementation.

If this options is set the BayesStore::SQL module will override the set username with the value given. This could be useful for implementing global or group
bayes databases.

bayes_use_hapaxes (default: 1)
Should the Bayesian classifier use hapaxes (words/tokens that occur only once) when classifying? This produces significantly better hit-rates, but increases
database size by about a factor of 8 to 10.

bayes_journal_max_size (default: 102400)
SpamAssassin will opportunistically sync the journal and the database. It will do so once a day, but will sync more often if the journal file size goes above
this setting, in bytes. If set to 0, opportunistic syncing will not occur.

bayes_expiry_max_db_size (default: 150000)
What should be the maximum size of the Bayes tokens database? When expiry occurs, the Bayes system will keep either 75% of the maximum value, or 100,000 tokens,
whichever has a larger value. 150,000 tokens is roughly equivalent to a 8Mb database file.

bayes_auto_expire (default: 1)
If enabled, the Bayes system will try to automatically expire old tokens from the database. Auto-expiry occurs when the number of tokens in the database
surpasses the bayes_expiry_max_db_size value.

bayes_learn_to_journal (default: 0)
If this option is set, whenever SpamAssassin does Bayes learning, it will put the information into the journal instead of directly into the database. This lowers
contention for locking the database to execute an update, but will also cause more access to the journal and cause a delay before the updates are actually
committed to the Bayes database.

MISCELLANEOUS OPTIONS
lock_method type
Select the file-locking method used to protect database files on-disk. By default, SpamAssassin uses an NFS-safe locking method on UNIX; however, if you are sure
that the database files you'll be using for Bayes and AWL storage will never be accessed over NFS, a non-NFS-safe locking system can be selected.

This will be quite a bit faster, but may risk file corruption if the files are ever accessed by multiple clients at once, and one or more of them is accessing
them through an NFS filesystem.

Note that different platforms require different locking systems.

The supported locking systems for "type" are as follows:

nfssafe - an NFS-safe locking system
flock - simple UNIX "flock()" locking
win32 - Win32 locking using "sysopen (..., O_CREAT|O_EXCL)".

nfssafe and flock are only available on UNIX, and win32 is only available on Windows. By default, SpamAssassin will choose either nfssafe or win32 depending on
the platform in use.

fold_headers ( 0 | 1 ) (default: 1)
By default, headers added by SpamAssassin will be whitespace folded. In other words, they will be broken up into multiple lines instead of one very long one and
each other line will have a tabulator prepended to mark it as a continuation of the preceding one.

The automatic wrapping can be disabled here. Note that this can generate very long lines.

report_safe_copy_headers header_name ...
If using "report_safe", a few of the headers from the original message are copied into the wrapper header (From, To, Cc, Subject, Date, etc.) If you want to have
other headers copied as well, you can add them using this option. You can specify multiple headers on the same line, separated by spaces, or you can just use
multiple lines.

envelope_sender_header Name-Of-Header
SpamAssassin will attempt to discover the address used in the 'MAIL FROM:' phase of the SMTP transaction that delivered this message, if this data has been made
available by the SMTP server. This is used in the "EnvelopeFrom" pseudo-header, and for various rules such as SPF checking.

By default, various MTAs will use different headers, such as the following:

X-Envelope-From
Envelope-Sender
X-Sender
Return-Path

SpamAssassin will attempt to use these, if some heuristics (such as the header placement in the message, or the absence of fetchmail signatures) appear to
indicate that they are safe to use. However, it may choose the wrong headers in some mailserver configurations. (More discussion of this can be found in bug
2142 and bug 4747 in the SpamAssassin BugZilla.)

To avoid this heuristic failure, the "envelope_sender_header" setting may be helpful. Name the header that your MTA adds to messages containing the address used
at the MAIL FROM step of the SMTP transaction.

If the header in question contains "<" or ">" characters at the start and end of the email address in the right-hand side, as in the SMTP transaction, these will
be stripped.

If the header is not found in a message, or if it's value does not contain an "@" sign, SpamAssassin will issue a warning in the logs and fall back to its default
heuristics.

(Note for MTA developers: we would prefer if the use of a single header be avoided in future, since that precludes 'downstream' spam scanning.
"http://wiki.apache.org/spamassassin/EnvelopeSenderInReceived" details a better proposal, storing the envelope sender at each hop in the "Received" header.)

example:

envelope_sender_header X-SA-Exim-Mail-From

describe SYMBOLIC_TEST_NAME description ...
Used to describe a test. This text is shown to users in the detailed report.

Note that test names which begin with '__' are reserved for meta-match sub-rules, and are not scored or listed in the 'tests hit' reports.

Also note that by convention, rule descriptions should be limited in length to no more than 50 characters.

report_charset CHARSET (default: unset)
Set the MIME Content-Type charset used for the text/plain report which is attached to spam mail messages.

report ...some text for a report...
Set the report template which is attached to spam mail messages. See the "10_default_prefs.cf" configuration file in "/usr/share/spamassassin" for an example.

If you change this, try to keep it under 78 columns. Each "report" line appends to the existing template, so use "clear_report_template" to restart.

Tags can be included as explained above.

clear_report_template
Clear the report template.

report_contact ...text of contact address...
Set what _CONTACTADDRESS_ is replaced with in the above report text. By default, this is 'the administrator of that system', since the hostname of the system the
scanner is running on is also included.

report_hostname ...hostname to use...
Set what _HOSTNAME_ is replaced with in the above report text. By default, this is determined dynamically as whatever the host running SpamAssassin calls itself.

unsafe_report ...some text for a report...
Set the report template which is attached to spam mail messages which contain a non-text/plain part. See the "10_default_prefs.cf" configuration file in
"/usr/share/spamassassin" for an example.

Each "unsafe-report" line appends to the existing template, so use "clear_unsafe_report_template" to restart.

Tags can be used in this template (see above for details).

clear_unsafe_report_template
Clear the unsafe_report template.

RULE DEFINITIONS AND PRIVILEGED SETTINGS
These settings differ from the ones above, in that they are considered 'privileged'. Only users running "spamassassin" from their procmailrc's or forward files, or
sysadmins editing a file in "/etc/spamassassin", can use them. "spamd" users cannot use them in their "user_prefs" files, for security and efficiency reasons,
unless "allow_user_rules" is enabled (and then, they may only add rules from below).

allow_user_rules ( 0 | 1 ) (default: 0)
This setting allows users to create rules (and only rules) in their "user_prefs" files for use with "spamd". It defaults to off, because this could be a severe
security hole. It may be possible for users to gain root level access if "spamd" is run as root. It is NOT a good idea, unless you have some other way of ensuring
that users' tests are safe. Don't use this unless you are certain you know what you are doing. Furthermore, this option causes spamassassin to recompile all the
tests each time it processes a message for a user with a rule in his/her "user_prefs" file, which could have a significant effect on server load. It is not
recommended.

Note that it is not currently possible to use "allow_user_rules" to modify an existing system rule from a "user_prefs" file with "spamd".

redirector_pattern /pattern/modifiers
A regex pattern that matches both the redirector site portion, and the target site portion of a URI.

Note: The target URI portion must be surrounded in parentheses and
no other part of the pattern may create a backreference.

Example: http://chkpt.zdnet.com/chkpt/whatever/spammer.domain/yo/dude

redirector_pattern /^https?:\/\/(?:opt\.)?chkpt\.zdnet\.com\/chkpt\/\w+\/(.*)$/i

header SYMBOLIC_TEST_NAME header op /pattern/modifiers [if-unset: STRING]
Define a test. "SYMBOLIC_TEST_NAME" is a symbolic test name, such as 'FROM_ENDS_IN_NUMS'. "header" is the name of a mail header, such as 'Subject', 'To', etc.

Appending ":raw" to the header name will inhibit decoding of quoted-printable or base-64 encoded strings.

Appending ":addr" to the header name will cause everything except the first email address to be removed from the header. For example, all of the following will
result in "example@foo":

example@foo
example@foo (Foo Blah)
example@foo, example@bar
display: example@foo (Foo Blah), example@bar ;
Foo Blah <example@foo>
"Foo Blah" <example@foo>
"'Foo Blah'" <example@foo>

Appending ":name" to the header name will cause everything except the first real name to be removed from the header. For example, all of the following will
result in "Foo Blah"

example@foo (Foo Blah)
example@foo (Foo Blah), example@bar
display: example@foo (Foo Blah), example@bar ;
Foo Blah <example@foo>
"Foo Blah" <example@foo>
"'Foo Blah'" <example@foo>

There are several special pseudo-headers that can be specified:

"ALL" can be used to mean the text of all the message's headers.
"ToCc" can be used to mean the contents of both the 'To' and 'Cc' headers.
"EnvelopeFrom" is the address used in the 'MAIL FROM:' phase of the SMTP transaction that delivered this message, if this data has been made available by the SMTP
server. See "envelope_sender_header" for more information on how to set this.
"MESSAGEID" is a symbol meaning all Message-Id's found in the message; some mailing list software moves the real 'Message-Id' to 'Resent-Message-Id' or
'X-Message-Id', then uses its own one in the 'Message-Id' header. The value returned for this symbol is the text from all 3 headers, separated by newlines.
"X-Spam-Relays-Untrusted", "X-Spam-Relays-Trusted", "X-Spam-Relays-Internal" and "X-Spam-Relays-External" represent a portable, pre-parsed representation of the
message's network path, as recorded in the Received headers, divided into 'trusted' vs 'untrusted' and 'internal' vs 'external' sets. See
"http://wiki.apache.org/spamassassin/TrustedRelays" for more details.

"op" is either "=~" (contains regular expression) or "!~" (does not contain regular expression), and "pattern" is a valid Perl regular expression, with
"modifiers" as regexp modifiers in the usual style. Note that multi-line rules are not supported, even if you use "x" as a modifier. Also note that the "#"
character must be escaped ("\#") or else it will be considered to be the start of a comment and not part of the regexp.

If the "[if-unset: STRING]" tag is present, then "STRING" will be used if the header is not found in the mail message.

Test names must not start with a number, and must contain only alphanumerics and underscores. It is suggested that lower-case characters not be used, and names
have a length of no more than 22 characters, as an informal convention. Dashes are not allowed.

Note that test names which begin with '__' are reserved for meta-match sub-rules, and are not scored or listed in the 'tests hit' reports. Test names which begin
with 'T_' are reserved for tests which are undergoing QA, and these are given a very low score.

If you add or modify a test, please be sure to run a sanity check afterwards by running "spamassassin --lint". This will avoid confusing error messages, or other
tests being skipped as a side-effect.

header SYMBOLIC_TEST_NAME exists:name_of_header
Define a header existence test. "name_of_header" is the name of a header to test for existence. This is just a very simple version of the above header tests.

header SYMBOLIC_TEST_NAME eval:name_of_eval_method([arguments])
Define a header eval test. "name_of_eval_method" is the name of a method on the "Mail::SpamAssassin::EvalTests" object. "arguments" are optional arguments to
the function call.

header SYMBOLIC_TEST_NAME eval:check_rbl('set', 'zone' [, 'sub-test'])
Check a DNSBL (a DNS blacklist or whitelist). This will retrieve Received: headers from the message, extract the IP addresses, select which ones are 'untrusted'
based on the "trusted_networks" logic, and query that DNSBL zone. There's a few things to note:

duplicated or private IPs
Duplicated IPs are only queried once and reserved IPs are not queried. Private IPs are those listed in <http://www.iana.org/assignments/ipv4-address-space>,
<http://duxcw.com/faq/network/privip.htm>, <http://duxcw.com/faq/network/autoip.htm>, or <ftp://ftp.rfc-editor.org/in-notes/rfc3330.txt> as private.

the 'set' argument
This is used as a 'zone ID'. If you want to look up a multiple-meaning zone like NJABL or SORBS, you can then query the results from that zone using it; but
all check_rbl_sub() calls must use that zone ID.

Also, if more than one IP address gets a DNSBL hit for a particular rule, it does not affect the score because rules only trigger once per message.

the 'zone' argument
This is the root zone of the DNSBL, ending in a period.

the 'sub-test' argument
This optional argument behaves the same as the sub-test argument in "check_rbl_sub()" below.

selecting all IPs except for the originating one
This is accomplished by placing '-notfirsthop' at the end of the set name. This is useful for querying against DNS lists which list dialup IP addresses; the
first hop may be a dialup, but as long as there is at least one more hop, via their outgoing SMTP server, that's legitimate, and so should not gain points.
If there is only one hop, that will be queried anyway, as it should be relaying via its outgoing SMTP server instead of sending directly to your MX (mail
exchange).

selecting IPs by whether they are trusted
When checking a 'nice' DNSBL (a DNS whitelist), you cannot trust the IP addresses in Received headers that were not added by trusted relays. To test the
first IP address that can be trusted, place '-firsttrusted' at the end of the set name. That should test the IP address of the relay that connected to the
most remote trusted relay.

Note that this requires that SpamAssassin know which relays are trusted. For simple cases, SpamAssassin can make a good estimate. For complex cases, you may
get better results by setting "trusted_networks" manually.

Selecting just the last external IP
By using '-lastexternal' at the end of the set name, you can select only the external host that connected to your internal network, or at least the last
external host with a public IP.

header SYMBOLIC_TEST_NAME eval:check_rbl_txt('set', 'zone')
Same as check_rbl(), except querying using IN TXT instead of IN A records. If the zone supports it, it will result in a line of text describing why the IP is
listed, typically a hyperlink to a database entry.

header SYMBOLIC_TEST_NAME eval:check_rbl_sub('set', 'sub-test')
Create a sub-test for 'set'. If you want to look up a multi-meaning zone like relays.osirusoft.com, you can then query the results from that zone using the zone
ID from the original query. The sub-test may either be an IPv4 dotted address for RBLs that return multiple A records or a non-negative decimal number to specify
a bitmask for RBLs that return a single A record containing a bitmask of results, a SenderBase test beginning with "sb:", or (if none of the preceding options
seem to fit) a regular expression.

Note: the set name must be exactly the same for as the main query rule, including selections like '-notfirsthop' appearing at the end of the set name.

body SYMBOLIC_TEST_NAME /pattern/modifiers
Define a body pattern test. "pattern" is a Perl regular expression. Note: as per the header tests, "#" must be escaped ("\#") or else it is considered the
beginning of a comment.

The 'body' in this case is the textual parts of the message body; any non-text MIME parts are stripped, and the message decoded from Quoted-Printable or
Base-64-encoded format if necessary. The message Subject header is considered part of the body and becomes the first paragraph when running the rules. All HTML
tags and line breaks will be removed before matching.

body SYMBOLIC_TEST_NAME eval:name_of_eval_method([args])
Define a body eval test. See above.

uri SYMBOLIC_TEST_NAME /pattern/modifiers
Define a uri pattern test. "pattern" is a Perl regular expression. Note: as per the header tests, "#" must be escaped ("\#") or else it is considered the
beginning of a comment.

The 'uri' in this case is a list of all the URIs in the body of the email, and the test will be run on each and every one of those URIs, adjusting the score if a
match is found. Use this test instead of one of the body tests when you need to match a URI, as it is more accurately bound to the start/end points of the URI,
and will also be faster.

rawbody SYMBOLIC_TEST_NAME /pattern/modifiers
Define a raw-body pattern test. "pattern" is a Perl regular expression. Note: as per the header tests, "#" must be escaped ("\#") or else it is considered the
beginning of a comment.

The 'raw body' of a message is the raw data inside all textual parts. The text will be decoded from base64 or quoted-printable encoding, but HTML tags and line
breaks will still be present. The pattern will be applied line-by-line.

rawbody SYMBOLIC_TEST_NAME eval:name_of_eval_method([args])
Define a raw-body eval test. See above.

full SYMBOLIC_TEST_NAME /pattern/modifiers
Define a full message pattern test. "pattern" is a Perl regular expression. Note: as per the header tests, "#" must be escaped ("\#") or else it is considered
the beginning of a comment.

The full message is the pristine message headers plus the pristine message body, including all MIME data such as images, other attachments, MIME boundaries, etc.

full SYMBOLIC_TEST_NAME eval:name_of_eval_method([args])
Define a full message eval test. See above.

meta SYMBOLIC_TEST_NAME boolean expression
Define a boolean expression test in terms of other tests that have been hit or not hit. For example:

meta META1 TEST1 && !(TEST2 || TEST3)

Note that English language operators ("and", "or") will be treated as rule names, and that there is no "XOR" operator.

meta SYMBOLIC_TEST_NAME boolean arithmetic expression
Can also define a boolean arithmetic expression in terms of other tests, with an unhit test having the value "0" and a hit test having a nonzero value. The value
of a hit meta test is that of its arithmetic expression. The value of a hit eval test is that returned by its method. The value of a hit header, body, rawbody,
uri, or full test which has the "multiple" tflag is the number of times the test hit. The value of any other type of hit test is "1".

For example:

meta META2 (3 * TEST1 - 2 * TEST2) > 0

Note that Perl builtins and functions, like "abs()", can't be used, and will be treated as rule names.

If you want to define a meta-rule, but do not want its individual sub-rules to count towards the final score unless the entire meta-rule matches, give the sub-
rules names that start with '__' (two underscores). SpamAssassin will ignore these for scoring.

tflags SYMBOLIC_TEST_NAME [ {net|nice|learn|userconf|noautolearn|multiple} ]
Used to set flags on a test. These flags are used in the score-determination back end system for details of the test's behaviour. Please see "bayes_auto_learn"
for more information about tflag interaction with those systems. The following flags can be set:

net The test is a network test, and will not be run in the mass checking system or if -L is used, therefore its score should not be modified.

nice
The test is intended to compensate for common false positives, and should be assigned a negative score.

userconf
The test requires user configuration before it can be used (like language- specific tests).

learn
The test requires training before it can be used.

noautolearn
The test will explicitly be ignored when calculating the score for learning systems.

multiple
The test will be evaluated multiple times, for use with meta rules. Only affects header, body, rawbody, uri, and full tests.

priority SYMBOLIC_TEST_NAME n
Assign a specific priority to a test. All tests, except for DNS and Meta tests, are run in increasing priority value order (negative priority values are run
before positive priority values). The default test priority is 0 (zero).

The values <-99999999999999> and <-99999999999998> have a special meaning internally, and should not be used.

ADMINISTRATOR SETTINGS
These settings differ from the ones above, in that they are considered 'more privileged' -- even more than the ones in the PRIVILEGED SETTINGS section. No matter
what "allow_user_rules" is set to, these can never be set from a user's "user_prefs" file when spamc/spamd is being used. However, all settings can be used by local
programs run directly by the user.

version_tag string
This tag is appended to the SA version in the X-Spam-Status header. You should include it when modify your ruleset, especially if you plan to distribute it. A
good choice for string is your last name or your initials followed by a number which you increase with each change.

The version_tag will be lowercased, and any non-alphanumeric or period character will be replaced by an underscore.

e.g.

version_tag myrules1 # version=2.41-myrules1

test SYMBOLIC_TEST_NAME (ok|fail) Some string to test against
Define a regression testing string. You can have more than one regression test string per symbolic test name. Simply specify a string that you wish the test to
match.

These tests are only run as part of the test suite - they should not affect the general running of SpamAssassin.

rbl_timeout t [t_min] [zone] (default: 15 3)
All DNS queries are made at the beginning of a check and we try to read the results at the end. This value specifies the maximum period of time (in seconds) to
wait for an DNS query. If most of the DNS queries have succeeded for a particular message, then SpamAssassin will not wait for the full period to avoid wasting
time on unresponsive server(s), but will shrink the timeout according to a percentage of queries already completed. As the number of queries remaining approaches
0, the timeout value will gradually approach a t_min value, which is an optional second parameter and defaults to 0.2 * t. If t is smaller than t_min, the
initial timeout is set to t_min. Here is a chart of queries remaining versus the timeout in seconds, for the default 15 second / 3 second timeout setting:

queries left 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%
timeout 15 14.9 14.5 13.9 13.1 12.0 10.7 9.1 7.3 5.3 3

For example, if 20 queries are made at the beginning of a message check and 16 queries have returned (leaving 20%), the remaining 4 queries should finish within
7.3 seconds since their query started or they will be timed out. Note that timed out queries are only aborted when there is nothing else left for SpamAssassin to
do - long evaluation of other rules may grant queries additional time.

If a parameter 'zone' is specified (it must end with a letter, which distinguishes it from other numeric parametrs), then the setting only applies to DNS queries
against the specified DNS domain (host, domain or RBL (sub)zone). Matching is case-insensitive, the actual domain may be a subdomain of the specified zone.

util_rb_tld tld1 tld2 ...
This option allows the addition of new TLDs to the RegistrarBoundaries code. Updates to the list usually happen when new versions of SpamAssassin are released,
but sometimes it's necessary to add in new TLDs faster than a release can occur. TLDs include things like com, net, org, etc.

util_rb_2tld 2tld-1.tld 2tld-2.tld ...
This option allows the addition of new 2nd-level TLDs (2TLD) to the RegistrarBoundaries code. Updates to the list usually happen when new versions of
SpamAssassin are released, but sometimes it's necessary to add in new 2TLDs faster than a release can occur. 2TLDs include things like co.uk, fed.us, etc.

bayes_path /path/filename (default: ~/.spamassassin/bayes)
This is the directory and filename for Bayes databases. Several databases will be created, with this as the base directory and filename, with "_toks", "_seen",
etc. appended to the base. The default setting results in files called "~/.spamassassin/bayes_seen", "~/.spamassassin/bayes_toks", etc.

By default, each user has their own in their "~/.spamassassin" directory with mode 0700/0600. For system-wide SpamAssassin use, you may want to reduce disk space
usage by sharing this across all users. However, Bayes appears to be more effective with individual user databases.

bayes_file_mode (default: 0700)
The file mode bits used for the Bayesian filtering database files.

Make sure you specify this using the 'x' mode bits set, as it may also be used to create directories. However, if a file is created, the resulting file will not
have any execute bits set (the umask is set to 111).

bayes_store_module Name::Of::BayesStore::Module
If this option is set, the module given will be used as an alternate to the default bayes storage mechanism. It must conform to the published storage
specification (see Mail::SpamAssassin::BayesStore). For example, set this to Mail::SpamAssassin::BayesStore::SQL to use the generic SQL storage module.

bayes_sql_dsn DBI::databasetype:databasename:hostname:port
Used for BayesStore::SQL storage implementation.

This option give the connect string used to connect to the SQL based Bayes storage.

bayes_sql_username
Used by BayesStore::SQL storage implementation.

This option gives the username used by the above DSN.

bayes_sql_password
Used by BayesStore::SQL storage implementation.

This option gives the password used by the above DSN.

bayes_sql_username_authorized ( 0 | 1 ) (default: 0)
Whether to call the services_authorized_for_username plugin hook in BayesSQL. If the hook does not determine that the user is allowed to use bayes or is invalid
then then database will not be initialized.

NOTE: By default the user is considered invalid until a plugin returns a true value. If you enable this, but do not have a proper plugin loaded, all users will
turn up as invalid.

The username passed into the plugin can be affected by the bayes_sql_override_username config option.

user_scores_dsn DBI:databasetype:databasename:hostname:port
If you load user scores from an SQL database, this will set the DSN used to connect. Example: "DBI:mysql:spamassassin:localhost"

If you load user scores from an LDAP directory, this will set the DSN used to connect. You have to write the DSN as an LDAP URL, the components being the host and
port to connect to, the base DN for the seasrch, the scope of the search (base, one or sub), the single attribute being the multivalued attribute used to hold the
configuration data (space separated pairs of key and value, just as in a file) and finally the filter being the expression used to filter out the wanted username.
Note that the filter expression is being used in a sprintf statement with the username as the only parameter, thus is can hold a single __USERNAME__ expression.
This will be replaced with the username.

Example: "ldap://localhost:389/dc=koehntopp,dc=de?saconfig?uid=__USERNAME__"

user_scores_sql_username username
The authorized username to connect to the above DSN.

user_scores_sql_password password
The password for the database username, for the above DSN.

user_scores_sql_custom_query query
This option gives you the ability to create a custom SQL query to retrieve user scores and preferences. In order to work correctly your query should return two
values, the preference name and value, in that order. In addition, there are several "variables" that you can use as part of your query, these variables will be
substituted for the current values right before the query is run. The current allowed variables are:

_TABLE_
The name of the table where user scores and preferences are stored. Currently hardcoded to userpref, to change this value you need to create a new custom
query with the new table name.

_USERNAME_
The current user's username.

_MAILBOX_
The portion before the @ as derived from the current user's username.

_DOMAIN_
The portion after the @ as derived from the current user's username, this value may be null.

The query must be one one continuous line in order to parse correctly.

Here are several example queries, please note that these are broken up for easy reading, in your config it should be one continuous line.

Current default query:
"SELECT preference, value FROM _TABLE_ WHERE username = _USERNAME_ OR username = '@GLOBAL' ORDER BY username ASC"

Use global and then domain level defaults:
"SELECT preference, value FROM _TABLE_ WHERE username = _USERNAME_ OR username = '@GLOBAL' OR username = '@~'||_DOMAIN_ ORDER BY username ASC"

Maybe global prefs should override user prefs:
"SELECT preference, value FROM _TABLE_ WHERE username = _USERNAME_ OR username = '@GLOBAL' ORDER BY username DESC"

user_scores_ldap_username
This is the Bind DN used to connect to the LDAP server. It defaults to the empty string (""), allowing anonymous binding to work.

Example: "cn=master,dc=koehntopp,dc=de"

user_scores_ldap_password
This is the password used to connect to the LDAP server. It defaults to the empty string ("").

loadplugin PluginModuleName [/path/module.pm]
Load a SpamAssassin plugin module. The "PluginModuleName" is the perl module name, used to create the plugin object itself.

"/path/to/module.pm" is the file to load, containing the module's perl code; if it's specified as a relative path, it's considered to be relative to the current
configuration file. If it is omitted, the module will be loaded using perl's search path (the @INC array).

See "Mail::SpamAssassin::Plugin" for more details on writing plugins.
tryplugin PluginModuleName [/path/module.pm]
Same as "loadplugin", but silently ignored if the .pm file cannot be found in the filesystem.

PREPROCESSING OPTIONS
include filename
Include configuration lines from "filename". Relative paths are considered relative to the current configuration file or user preferences file.

if (conditional perl expression)
Used to support conditional interpretation of the configuration file. Lines between this and a corresponding "else" or "endif" line, will be ignored unless the
conditional expression evaluates as true (in the perl sense; that is, defined and non-0).

The conditional accepts a limited subset of perl for security -- just enough to perform basic arithmetic comparisons. The following input is accepted:

numbers, whitespace, arithmetic operations and grouping
Namely these characters and ranges:

( ) - + * / _ . , < = > ! ~ 0-9 whitespace

version
This will be replaced with the version number of the currently-running SpamAssassin engine. Note: The version used is in the internal SpamAssassin version
format which is "x.yyyzzz", where x is major version, y is minor version, and z is maintenance version. So 3.0.0 is 3.000000, and 3.4.80 is 3.004080.

plugin(Name::Of::Plugin)
This is a function call that returns 1 if the plugin named "Name::Of::Plugin" is loaded, or "undef" otherwise.

If the end of a configuration file is reached while still inside a "if" scope, a warning will be issued, but parsing will restart on the next file.

For example:

if (version > 3.000000)
header MY_FOO ...
endif

loadplugin MyPlugin plugintest.pm

if plugin (MyPlugin)
header MY_PLUGIN_FOO eval:check_for_foo()
score MY_PLUGIN_FOO 0.1
endif

ifplugin PluginModuleName
An alias for "if plugin(PluginModuleName)".
else
Used to support conditional interpretation of the configuration file. Lines between this and a corresponding "endif" line, will be ignored unless the conditional
expression evaluates as false (in the perl sense; that is, not defined and 0).

require_version n.nnnnnn
Indicates that the entire file, from this line on, requires a certain version of SpamAssassin to run. If a different (older or newer) version of SpamAssassin
tries to read the configuration from this file, it will output a warning instead, and ignore it.

Note: The version used is in the internal SpamAssassin version format which is "x.yyyzzz", where x is major version, y is minor version, and z is maintenance
version. So 3.0.0 is 3.000000, and 3.4.80 is 3.004080.

TEMPLATE TAGS
The following "tags" can be used as placeholders in certain options. They will be replaced by the corresponding value when they are used.

Some tags can take an argument (in parentheses). The argument is optional, and the default is shown below.

_YESNOCAPS_ "YES"/"NO" for is/isn't spam
_YESNO_ "Yes"/"No" for is/isn't spam
_SCORE(PAD)_ message score, if PAD is included and is either spaces or
zeroes, then pad scores with that many spaces or zeroes
(default, none) ie: _SCORE(0)_ makes 2.4 become 02.4,
_SCORE(00)_ is 002.4. 12.3 would be 12.3 and 012.3
respectively.
_REQD_ message threshold
_VERSION_ version (eg. 3.0.0 or 3.1.0-r26142-foo1)
_SUBVERSION_ sub-version/code revision date (eg. 2004-01-10)
_HOSTNAME_ hostname of the machine the mail was processed on
_REMOTEHOSTNAME_ hostname of the machine the mail was sent from, only
available with spamd
_REMOTEHOSTADDR_ ip address of the machine the mail was sent from, only
available with spamd
_BAYES_ bayes score
_TOKENSUMMARY_ number of new, neutral, spammy, and hammy tokens found
_BAYESTC_ number of new tokens found
_BAYESTCLEARNED_ number of seen tokens found
_BAYESTCSPAMMY_ number of spammy tokens found
_BAYESTCHAMMY_ number of hammy tokens found
_HAMMYTOKENS(N)_ the N most significant hammy tokens (default, 5)
_SPAMMYTOKENS(N)_ the N most significant spammy tokens (default, 5)
_DATE_ rfc-2822 date of scan
_STARS(*)_ one "*" (use any character) for each full score point
(note: limited to 50 'stars')
_RELAYSTRUSTED_ relays used and deemed to be trusted (see the
'X-Spam-Relays-Trusted' pseudo-header)
_RELAYSUNTRUSTED_ relays used that can not be trusted (see the
'X-Spam-Relays-Untrusted' pseudo-header)
_RELAYSINTERNAL_ relays used and deemed to be internal (see the
'X-Spam-Relays-Internal' pseudo-header)
_RELAYSEXTERNAL_ relays used and deemed to be external (see the
'X-Spam-Relays-External' pseudo-header)
_LASTEXTERNALIP_ IP address of client in the external-to-internal
SMTP handover
_LASTEXTERNALRDNS_ reverse-DNS of client in the external-to-internal
SMTP handover
_LASTEXTERNALHELO_ HELO string used by client in the external-to-internal
SMTP handover
_AUTOLEARN_ autolearn status ("ham", "no", "spam", "disabled",
"failed", "unavailable")
_AUTOLEARNSCORE_ portion of message score used by autolearn
_TESTS(,)_ tests hit separated by "," (or other separator)
_TESTSSCORES(,)_ as above, except with scores appended (eg. AWL=-3.0,...)
_SUBTESTS(,)_ subtests (start with "__") hit separated by ","
(or other separator)
_DCCB_ DCC's "Brand"
_DCCR_ DCC's results
_PYZOR_ Pyzor results
_RBL_ full results for positive RBL queries in DNS URI format
_LANGUAGES_ possible languages of mail
_PREVIEW_ content preview
_REPORT_ terse report of tests hit (for header reports)
_SUMMARY_ summary of tests hit for standard report (for body reports)
_CONTACTADDRESS_ contents of the 'report_contact' setting
_HEADER(NAME)_ includes the value of a message header. value is the same
as is found for header rules (see elsewhere in this doc)

If a tag reference uses the name of a tag which is not in this list or defined by a loaded plugin, the reference will be left intact and not replaced by any value.

The "HAMMYTOKENS" and "SPAMMYTOKENS" tags have an optional second argument which specifies a format. See the HAMMYTOKENS/SPAMMYTOKENS TAG FORMAT section, below, for
details.

HAMMYTOKENS/SPAMMYTOKENS TAG FORMAT
The "HAMMYTOKENS" and "SPAMMYTOKENS" tags have an optional second argument which specifies a format: "_SPAMMYTOKENS(N,FMT)_", "_HAMMYTOKENS(N,FMT)_" The following
formats are available:

short
Only the tokens themselves are listed. For example, preference file entry:

"add_header all Spammy _SPAMMYTOKENS(2,short)_"

Results in message header:

"X-Spam-Spammy: remove.php, UD:jpg"

Indicating that the top two spammy tokens found are "remove.php" and "UD:jpg". (The token itself follows the last colon, the text before the colon indicates
something about the token. "UD" means the token looks like it might be part of a domain name.)

compact
The token probability, an abbreviated declassification distance (see example), and the token are listed. For example, preference file entry:

"add_header all Spammy _SPAMMYTOKENS(2,compact)_"

Results in message header:

"0.989-6--remove.php, 0.988-+--UD:jpg"

Indicating that the probabilities of the top two tokens are 0.989 and 0.988, respectively. The first token has a declassification distance of 6, meaning that if
the token had appeared in at least 6 more ham messages it would not be considered spammy. The "+" for the second token indicates a declassification distance
greater than 9.

long
Probability, declassification distance, number of times seen in a ham message, number of times seen in a spam message, age and the token are listed.

For example, preference file entry:

"add_header all Spammy _SPAMMYTOKENS(2,long)_"

Results in message header:

"X-Spam-Spammy: 0.989-6--0h-4s--4d--remove.php, 0.988-33--2h-25s--1d--UD:jpg"

In addition to the information provided by the compact option, the long option shows that the first token appeared in zero ham messages and four spam messages,
and that it was last seen four days ago. The second token appeared in two ham messages, 25 spam messages and was last seen one day ago. (Unlike the "compact"
option, the long option shows declassification distances that are greater than 9.)

LOCALI[SZ]ATION
A line starting with the text "lang xx" will only be interpreted if the user is in that locale, allowing test descriptions and templates to be set for that language.

The locales string should specify either both the language and country, e.g. "lang pt_BR", or just the language, e.g. "lang de".

SEE ALSO
"Mail::SpamAssassin" "spamassassin" "spamd"

perl v5.10.0 2009-10-06 Mail::SpamAssassin::Conf(3pm)
</pre>

[[Category: Email]]
[[Category: Spamassassin]]

Spamassassin

2010-03-06T21:12:41+00:00

← Older revision		Revision as of 21:12, 6 March 2010
Line 118:		Line 118:
	* [[Squirrelmail Spam Buttons]]		* [[Squirrelmail Spam Buttons]]
	* [[Generate Passlists from Sent Mail]]		* [[Generate Passlists from Sent Mail]]
		+	* [[Mail::Spamassassin::Conf]]

	== External Links ==		== External Links ==

MediaWiki:Sidebar

2010-03-06T20:47:02+00:00

docunext pushed to master at docunext/sinatra-xslview

2010-03-06T18:34:55+00:00

docunext committed da59634

docs

docunext pushed to master at docunext/sinatra-xslview

2010-03-06T18:32:48+00:00

docunext committed 29e8234

can now pass a hash of parameters. wicked awesome
docunext committed 9e07a38

renamed readme

Amazon CloudFront

2010-03-06T17:05:20+00:00

← Older revision		Revision as of 17:05, 6 March 2010
(2 intermediate revisions not shown)
Line 1:		Line 1:
	I'm giving Amazon CloudFront a try, might be interesting. What is it? Its a simple content-distribution network.		I'm giving Amazon CloudFront a try, might be interesting. What is it? Its a simple content-distribution network.

		+	Seems like it should be possible to limit access to files based on an HTTP referer (referrer) header, but apparently that's not possible. This would be useful to prevent only certain domains from including an image on a web page.

		+
		+	== Response HTTP Header ==
		+	<pre class="sh_sh">
		+	HTTP/1.0 200 OK
		+	x-amz-id-2: ZeJZmFcCbb6eBhCxhD0xiHHUgm63M0aa8UTeHSdaWUgEu79cl+x6pEOs2VOLPlle
		+	x-amz-request-id: 3C26F220A0EBB9D5
		+	Date: Sat, 06 Mar 2010 17:03:28 GMT
		+	Cache-Control: public,max-age=315360000
		+	Last-Modified: Sat, 06 Mar 2010 16:14:40 GMT
		+	Etag: "6f83726eb883261b509158ac61eee361"
		+	Content-Type: image/jpeg
		+	Content-Length: 18939
		+	Server: AmazonS3
		+	X-Cache: Miss from cloudfront
		+	X-Amz-Cf-Id: aca6c397bdef5198d8cf831ebf78e416bffdba20b357981c8187d51784fc8c4bedf0d05844e5e404,3b51b1e33c5afb907dc65330afcebd9e0cc88cd093af3dfc2529b2f9f31ac067540842e0676b4ef1
		+	Via: 1.0 ecf6abe40feed656b4a0843263468b70.cloudfront.net:11180 (CloudFront), 1.0 e45b1f1e171044cf488be122ea3ff12a.cloudfront.net:11180 (CloudFront)
		+	Connection: keep-alive
		+	</pre>
		+
		+	After clearing my local cache and requesting the file again:
		+	<pre>
		+	HTTP/1.0 200 OK
		+	x-amz-id-2: ZeJZmFcCbb6eBhCxhD0xiHHUgm63M0aa8UTeHSdaWUgEu79cl+x6pEOs2VOLPlle
		+	x-amz-request-id: 3C26F220A0EBB9D5
		+	Date: Sat, 06 Mar 2010 17:03:28 GMT
		+	Cache-Control: public,max-age=315360000
		+	Last-Modified: Sat, 06 Mar 2010 16:14:40 GMT
		+	Etag: "6f83726eb883261b509158ac61eee361"
		+	Content-Type: image/jpeg
		+	Content-Length: 18939
		+	Server: AmazonS3
		+	Age: 65
		+	X-Cache: Hit from cloudfront
		+	X-Amz-Cf-Id: 5d98835eb81951b4c15e329fc1e150ef1e0ab322ee94ca5f31483119ed433caa0464041ee77abc67
		+	Via: 1.0 ecf6abe40feed656b4a0843263468b70.cloudfront.net:11180 (CloudFront), 1.0 e45b1f1e171044cf488be122ea3ff12a.cloudfront.net:11180 (CloudFront)
		+	Connection: keep-alive
		+	</pre>
		+
		+	This works very much like a [[Varnish]] node.

	== External Links ==		== External Links ==

Amazon

2010-03-06T16:31:54+00:00

← Older revision		Revision as of 16:31, 6 March 2010
Line 2:		Line 2:

	* [[S3]]		* [[S3]]
		+	* [[Amazon CloudFront]]

-		+	[[Category: Technology Companies]]
-	[[Category:Technology Companies]]	+

Category:Amazon

2010-03-06T16:29:54+00:00

Created page with 'Amazon'

New page

[[Amazon]]

Amazon CloudFront

2010-03-06T16:28:00+00:00

Created page with 'I'm giving Amazon CloudFront a try, might be interesting. What is it? Its a simple content-distribution network. == External Links == * http://www.labnol.org/internet/setup-co...'

New page

I'm giving Amazon CloudFront a try, might be interesting. What is it? Its a simple content-distribution network.

== External Links ==
* http://www.labnol.org/internet/setup-content-delivery-network-with-amazon-s3-cloudfront/5446/

[[Category: Amazon]]

S3

2010-03-06T16:27:14+00:00

S3 Tools:

← Older revision		Revision as of 16:27, 6 March 2010
(4 intermediate revisions not shown)
Line 2:		Line 2:
	S3 is short or Simple Storage Service. Its offered by Amazon and is reasonably priced.		S3 is short or Simple Storage Service. Its offered by Amazon and is reasonably priced.

		+	== My Experience with Amazon S3 ==
		+	I've wanted to use Amazon's S3 time and time again, but for various reasons its never gained any traction for me. I get concerned about performance, control, and manageability.

		+	I always like to cache aggressively, too. The default headers for a request look like this:
		+
		+	<pre class="sh_log">
		+	HTTP/1.1 200 OK
		+	x-amz-id-2: GxdP9Jjq0hm1CzRqsspD898voGu6tSZiMrF102nujWGarp0IM5v5/iXsDpsia4Zy
		+	x-amz-request-id: 7C29E28AEDA8B257
		+	Date: Sat, 06 Mar 2010 15:50:40 GMT
		+	Last-Modified: Sat, 06 Jun 2009 01:10:17 GMT
		+	Etag: "6f83726eb883261b509158ac61eee361"
		+	Content-Type: image/jpeg
		+	Content-Length: 18939
		+	Server: AmazonS3
		+	</pre>
		+
		+	The last-modified and etag headers help with performance, and the S3 servers seem to have pretty good latency. Here's a subsequent forced request:
		+
		+	<pre class="sh_log">
		+	Response Headers
		+	x-amz-id-2 ERZ5FRSgrxQRNR/qdWaItwxNadXHAyK9MM09IbdrHYQ1b1C0tWAGlKKatJat66xB
		+	x-amz-request-id 7CE0E88ED8D462E9
		+	Date Sat, 06 Mar 2010 16:02:12 GMT
		+	Last-Modified Sat, 06 Jun 2009 01:10:17 GMT
		+	Etag "6f83726eb883261b509158ac61eee361"
		+	Content-Type image/jpeg
		+	Content-Length 18939
		+	Server AmazonS3
		+
		+	Request Headers
		+	Host www-sa.evenserver.com
		+	User-Agent Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.6) Gecko/20091216 Iceweasel/3.5.8 (like Firefox/3.5.8)
		+	Accept text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
		+	Accept-Language en-us,en;q=0.5
		+	Accept-Encoding gzip,deflate
		+	Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
		+	Keep-Alive 300
		+	Connection keep-alive
		+	If-Modified-Since Sat, 06 Jun 2009 01:10:17 GMT
		+	If-None-Match "6f83726eb883261b509158ac61eee361"
		+	Cache-Control max-age=0
		+	</pre>
		+
		+	That http request took 46ms of time. Not bad!
		+
		+	The headers can also be modified when changing access control:
		+
		+	<pre class="sh_sh">
		+	s3cmd --acl-public --add-header=Cache-Control:public,max-age=315360000 setacl s3://www-sa.evenserver.com/s/img/2008_informed_licensing.jpg
		+	</pre>
		+
		+	Resulting in this:
		+
		+	<pre class="sh_log">
		+	HTTP/1.1 200 OK
		+	x-amz-id-2: cx6ga6s5YBQX/UetB7w2u0d/ymIsA6FrrtfzdoC8NZhwxSY+ZQwxEU3N0e2eoG5G
		+	x-amz-request-id: 8DA7CD8BF9AAF751
		+	Date: Sat, 06 Mar 2010 16:15:46 GMT
		+	Cache-Control: public,max-age=315360000
		+	Last-Modified: Sat, 06 Mar 2010 16:14:40 GMT
		+	Etag: "6f83726eb883261b509158ac61eee361"
		+	Content-Type: image/jpeg
		+	Content-Length: 18939
		+	Server: AmazonS3
		+	</pre>
		+
		+	Much better!

	== S3 Tools ==		== S3 Tools ==
Line 9:		Line 76:
	* [[Apache2::S3]]		* [[Apache2::S3]]

-	[[Category:Storage]]	+	== Related Pages ==
-	[[Category:Cloud Computing]]	+	* [[Amazon CloudFront]]
		+
		+	== External Links ==
		+	* http://keakaj.com/wisdom/2007/07/27/cache-control-header-for-amazon-s3/
		+
		+	[[Category: Storage]]
		+	[[Category: Cloud Computing]]

Httping

2010-03-06T15:57:37+00:00

Created page with 'Httping is a cool little command line utility which, like Ping, produces some diagnostics about network connectivity but only over HTTP. Category: Networking'

New page

Httping is a cool little command line utility which, like [[Ping]], produces some diagnostics about network connectivity but only over [[HTTP]].

[[Category: Networking]]

Ping

2010-03-06T15:55:29+00:00

← Older revision		Revision as of 15:55, 6 March 2010
Line 2:		Line 2:


-	[[Category:Networking]]	+	== Related Pages ==
		+	* [[Httping]]
		+
		+	[[Category: Networking]]

Converting from Courier IMAP to Dovecot IMAP

2010-03-06T12:11:22+00:00

New page

<pre>
#!/usr/bin/perl
# cpanel12 - maildir-migrate Copyright(c) 2008 cPanel, Inc.
# All Rights Reserved.
# copyright@cpanel.net http://cpanel.net

# Based largely on courier-dovecot-migrate.pl v1.1.7
# Copyright 2008 Timo Sirainen

# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of the cPanel, Inc. nor the
# names of its contributors may be used to endorse or promote products
# derived from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY CPANEL, INC. "AS IS" AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL CPANEL, INC BE LIABLE FOR ANY
# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

use strict;
use warnings;
use Getopt::Long ();

# Key files in maildirs
my $courier_imap_uidfile = 'courierimapuiddb';
my $courier_pop3_uidfile = 'courierpop3dsizelist';
my $courier_subscriptions_file = 'courierimapsubscribed';
my $courier_keywords_dir = 'courierimapkeywords/';
my $courier_keywords_file = 'courierimapkeywords/:list';
my $dovecot_uidfile = 'dovecot-uidlist';
my $dovecot_uidvalidity_file = 'dovecot-uidvalidity';
my $dovecot_subscriptions_file = 'subscriptions';
my $dovecot_keywords_file = 'dovecot-keywords';

# Globals
my $do_conversion = 0;
my $quiet = 0;
my $recursive = 0;
my $overwrite = 0;

my $depth = 1;
my $maildir_subdirs = -1;
my $global_error_count = 0;
my $global_mailbox_count = 0;
my $global_user_count = 0;
my $uidlist_write_count = 0;
my $convert_to_dovecot = 0;
my $convert_to_courier = 0;
my $help = 0;
my $maildir_name = 'Maildir';

# Argument processing
my %opts = (
'convert' => \$do_conversion,
'quiet' => \$quiet,
'overwrite' => \$overwrite,
'recursive' => \$recursive,
'to-dovecot' => \$convert_to_dovecot,
'to-courier' => \$convert_to_courier,
'help' => \$help,
);

Getopt::Long::GetOptions(%opts);
usage() if $help;

my $mailroot = shift @ARGV || '.';

my $conversion_type;
if ( $convert_to_dovecot && $convert_to_courier ) {
print "It is not possible to convert to both Dovecot and Courier formats simultaneously!\n";
exit 1;
}
else {
print $do_conversion ? 'Converting' : 'Testing conversion' unless $quiet;

if ($convert_to_courier) {
print " to Courier format\n" unless $quiet;
$conversion_type = 'courier';
}
elsif ($convert_to_dovecot) {
print " to Dovecot format\n" unless $quiet;
$conversion_type = 'dovecot';
}
else {
print " based on newest timestamps\n" unless $quiet;
$conversion_type = 'auto';
}
}

# Check/Convert maildirs
print "Finding maildirs under $mailroot\n" if ( !$quiet );
if ( is_maildir($mailroot) ) {
check_maildir($mailroot);
}
elsif ( -d "$mailroot/$maildir_name" ) {
if ( !is_maildir("$mailroot/$maildir_name") ) {
print STDERR "$mailroot/$maildir_name doesn't seem to contain a valid Maildir\n";
}
else {
check_maildir("$mailroot/$maildir_name");
}
}
elsif ($recursive) {
if ( $depth > 0 || !userdir_check($mailroot) ) {
$depth-- if ( $depth > 0 );
if ( !depth_check( $mailroot, $depth ) ) {
print STDERR "No maildirs found\n";
exit;
}
}
}

# Totals
if ( !$quiet ) {
print "\nTotal: $global_mailbox_count mailboxes / $global_user_count users\n";
print " $global_error_count errors\n";

if ( !$do_conversion ) {
print "No actual conversion done, use --convert parameter\n";
}
else {
print "$uidlist_write_count $dovecot_uidfile files written\n";
}
print "\nWARNING: Badly done migration will cause your IMAP and/or POP3 clients to re-download all mails. Read http://wiki.dovecot.org/Migration carefully.\n";
}

sub scan_maildir {
my ( $dir, $map ) = @_;

my @scan_maildir_files;
if ( opendir my $scan_maildir_dh, $dir ) {
@scan_maildir_files = readdir($scan_maildir_dh);
closedir $scan_maildir_dh;
}
foreach my $real_filename (@scan_maildir_files) {
next if ( $real_filename eq "." || $real_filename eq ".." );

my $base_filename;
if ( $real_filename =~ /^([^:]+):2,/ ) {
$base_filename = $1;
}
else {
$base_filename = $real_filename;
}
$$map{$base_filename} = $real_filename;
}
}

sub read_courier_pop3 {
my ( $dir ) = @_;

my ( $pop3_uidv, $pop3_nextuid ) = ( -1, 0 );
my $filename_map = {};

my $f;
my $pop3_fname = "$dir/$courier_pop3_uidfile";
open( $f, $pop3_fname ) || die $!;
my $pop3_hdr = <$f>;
if ( $pop3_hdr =~ /^\/2 (\d+) (\d+)$/ ) {
# /2 <next uid> <uidvalidity>
$_ = <$f>;
}
elsif ( $pop3_hdr =~ /^\/1 (\d+)$/ ) {
# /1 <next uid>
$_ = <$f>;
}
elsif ( $pop3_hdr !~ /^\// ) {
# version 0: no UIDs
$_ = $pop3_hdr;
}
else {
print STDERR "$pop3_fname: Broken header: $pop3_hdr\n";
close $f;
return $filename_map;
}

my $line = 0;
do {
chomp $_;
$line++;

my ( $full_fname, $fsize, $uid, $uidv );

if ( /^([^ ]+) (\d+) (\d+):(\d+)$/ ) {
# v2
( $full_fname, $fsize, $uid, $uidv ) = ( $1, $2, $3, $4 );
}
elsif ( /^([^ ]+) (\d+) (\d+)$/ ) {
# v1
( $full_fname, $fsize, $uid ) = ( $1, $2, $3 );
$uidv = 0;
}
elsif ( /^([^ ]+) (\d+)$/ ) {
# v0
( $full_fname, $fsize ) = ( $1, $2 );
$uid = 0;
$uidv = 0;
}
else {
$global_error_count++;
print STDERR "$pop3_fname: Broken line: $_\n";
next;
}

# get base filename
my $fname = $full_fname;
$fname =~ s/^([^:]+).*$/$1/;

my $uidl = "";
if ( $uid == 0 ) {
# use filename
foreach (split(//, $fname)) {
if (ord($_) < 0x21 || ord($_) > 0x7e || $_ eq "'" || $_ eq '"' || $_ eq "+") {
$uidl .= sprintf("+%02X", ord($_));
} else {
$uidl .= $_;
}
}
}
elsif ($uidv != 0) {
$uidl = "UID$uid-$uidv";
}
else {
$uidl = "UID$uid";
}
$filename_map->{$fname} = [ 0, $uidl, $fsize, $full_fname, $line ];
} while (<$f>);
close $f;
return $filename_map;
}

sub read_courier_imap {
my ( $dir, $filename_map ) = @_;

# check if we can preserve IMAP UIDs
my $imap_fname = "$dir/$courier_imap_uidfile";
if ( !-f $imap_fname ) {
print "$imap_fname: OK\n" if ( !$quiet );
return;
}

my $f;
open( $f, $imap_fname ) || die $!;
my $imap_hdr = <$f>;
if ( $imap_hdr !~ /^1 (\d+) (\d+)$/ ) {
$global_error_count++;
print STDERR "$imap_fname: Broken header: $imap_hdr\n";
close $f;
return;
}
my ( $uidv, $nextuid ) = ( $1, $2 );

my %found_files;
my $found_files_looked_up = 0;
while (<$f>) {
chomp $_;

if (/^(\d+) (.*)$/) {
my ( $uid, $full_fname ) = ( $1, $2 );

# get the base filename
my $fname = $full_fname;
$fname =~ s/^([^:]+).*$/$1/;

if ( defined $filename_map->{$fname} ) {
$filename_map->{$fname}->[0] = $uid;
}
else {
# not in pop3 list
$filename_map->{$fname} = [ $uid, "", 0, $full_fname, 0 ];
}
$nextuid = $uid + 1 if ($uid >= $nextuid);
}
else {
$global_error_count++;
print STDERR "$imap_fname: Broken header\n";
}
}
close $f;

return ( $uidv, $nextuid, $filename_map );
}

sub write_dovecot_uidlist {
my ( $dir, $uidv, $nextuid, $owner_uid, $owner_gid, $filename_map ) = @_;

my $uidlist_fname = "$dir/$dovecot_uidfile";
if ( !$overwrite && -f $uidlist_fname ) {
print "$uidlist_fname already exists, not overwritten\n" if ( !$quiet );
return;
}
return if (scalar keys %{$filename_map} == 0);

return if ( !$do_conversion );

if ($uidv <= 0) {
# only pop3 UIDLs, generate a new uidvalidity
$uidv = time();
}

# POP3 clients may want to get POP3 UIDLs in the same order always.
# Preserve the order even if it causes IMAP UIDs to change.
my %order_map;
foreach my $fname ( keys %{$filename_map} ) {
my $order = $filename_map->{$fname}->[4];
$order_map{$order} = $fname if ($order != 0);
}
my $prev_uid = 0;
foreach my $order ( sort { $a <=> $b } keys %order_map ) {
my $file_ar = $filename_map->{ $order_map{$order} };
if ($file_ar->[0] == 0) {
$file_ar->[0] = $nextuid;
$nextuid++;
}

if ($file_ar->[0] <= $prev_uid) {
$file_ar->[0] = 0;
} else {
$prev_uid = $file_ar->[0];
}
}

my %uidlist_map;
foreach my $fname ( keys %{$filename_map} ) {
my $file_ar = $filename_map->{ $fname };
my $uid = $file_ar->[0];
if ($uid == 0 || ($file_ar->[4] == 0 && $uid <= $prev_uid)) {
# in pop3 list only, assign a new IMAP UID.
$uid = $nextuid;
$nextuid++;
}
$uidlist_map{$uid} = $fname;
}

open( my $dovecot_uidlist_fh, '>', $uidlist_fname ) || die $!;
print $dovecot_uidlist_fh "3 V$uidv N$nextuid\n";
foreach my $uid ( sort { $a <=> $b } keys %uidlist_map ) {
my $file_ar = $filename_map->{ $uidlist_map{$uid} };
print $dovecot_uidlist_fh "$uid ";
print $dovecot_uidlist_fh 'P' . $file_ar->[1] . ' ' if ( $file_ar->[1] ne "" );
print $dovecot_uidlist_fh 'W' . $file_ar->[2] . ' ' if ( $file_ar->[2] > 0 );
print $dovecot_uidlist_fh ':' . $file_ar->[3] . "\n";
}
close $dovecot_uidlist_fh;
chown $owner_uid, $owner_gid, $uidlist_fname;
$uidlist_write_count++;
}

sub convert_keywords {
my ( $dir, $owner_uid, $owner_gid ) = @_;

my $courier_mtime = ( stat("$dir/$courier_keywords_file") )[9] || 0;
my $dovecot_mtime = ( stat("$dir/$dovecot_keywords_file") )[9] || 0;

# No need to convert if there are no keywords files
return unless ( $courier_mtime || $dovecot_mtime );

# If we're doing auto-conversion, find the newest keywords file
my $convert_to = $conversion_type;
if ( $convert_to eq 'auto' ) {
$convert_to = $dovecot_mtime > $courier_mtime ? 'courier' : 'dovecot';
}

if ( $convert_to eq 'dovecot' ) {
# Courier to Dovecot keyword conversion
my $keyword_dir = "$dir/courierimapkeywords";
my $dovecot_keyfname = "$dir/dovecot-keywords";

if ( !-f "$keyword_dir/:list" ) {

# no keywords
return;
}

if ( !$overwrite && -f $dovecot_keyfname ) {
print "$dovecot_keyfname already exists, not overwritten\n" if ( !$quiet );
return;
}

my ( %keywords, %files );
my $f;
open( $f, "$keyword_dir/:list" ) || die $!;

# read keyword names
while (<$f>) {
chomp $_;

last if (/^$/);
$keywords{$_} = scalar keys %keywords;
}

# read filenames -> keywords mapping
while (<$f>) {
if (/([^:]+):([\d ]+)$/) {
my $fname = $1;
foreach ( sort { $a <=> $b } split( " ", $2 ) ) {
$files{$fname} .= chr( 97 + $_ );
}
}
else {
print STDERR "$keyword_dir/:list: Broken entry: $_\n";
}
}
close $f;

# read updates from the directory
my %updates;
my @update_files;
if ( opendir my $kw_dh, $keyword_dir ) {
@update_files = readdir($kw_dh);
closedir $kw_dh;
}
foreach (@update_files) {
next if ( $_ eq ":list" || $_ eq "." || $_ eq ".." );

my $fname = $_;
if (/^\.(\d+)\.(.*)$/) {
my ( $num, $base_fname ) = ( $1, $2 );
if ( !defined $updates{$fname} ) {
$updates{$fname} = $num;
}
else {
my $old = $updates{$fname};
if ( $old >= 0 && $num > $old ) {
$updates{$fname} = $num;
}
}
}
else {

# "fname" overrides .n.fnames
$updates{$fname} = -1;
}
}

# apply the updates
foreach ( keys %updates ) {
my $base_fname = $_;
my $num = $updates{$_};

my $fname;
if ( $num < 0 ) {
$fname = $base_fname;
}
else {
$fname = ".$num.$base_fname";
}

my @kw_list;
open( $f, "$keyword_dir/$fname" ) || next;
while (<$f>) {
chomp $_;
my $kw = $_;
my $idx;

if ( defined $keywords{$kw} ) {
$idx = $keywords{$kw};
}
else {
$idx = scalar keys %keywords;
$keywords{$kw} = $idx;
}
$kw_list[ scalar @kw_list ] = $idx;
}
close $f;

$files{$fname} = "";
foreach ( sort { $a <=> $b } @kw_list ) {
$files{$fname} .= chr( 97 + $_ );
}
}

return if ( !$do_conversion );

# write dovecot-keywords file
open( $f, ">$dovecot_keyfname" ) || die $!;
foreach ( sort { $keywords{$a} <=> $keywords{$b} } keys %keywords ) {
my $idx = $keywords{$_};
print $f "$idx $_\n";
}
close $f;
chown $owner_uid, $owner_gid, $dovecot_keyfname;

# update the maildir files
my $cur_dir = "$dir/cur";
my @cur_files;
if ( opendir my $cur_dir_dh, $cur_dir ) {
@cur_files = readdir($cur_dir_dh);
closedir $cur_dir_dh;
}
foreach (@cur_files) {
my $fname = $cur_dir . '/' . $_;

my ( $base_fname, $flags, $extra_flags );
if (/^([^:]+):2,([^,]*)(,.*)?$/) {
( $base_fname, $flags, $extra_flags ) = ( $1, $2, $3 );
$extra_flags = "" if ( !defined $extra_flags );
}
else {
$base_fname = $fname;
$flags = "";
$extra_flags = "";
}

if ( defined $files{$base_fname} ) {

# merge old and new flags
my %newflags;
foreach ( sort split( "", $files{$base_fname} ) ) {
$newflags{$_} = 1;
}
foreach ( sort split( "", $flags ) ) {
$newflags{$_} = 1;
}
$flags = "";
foreach ( sort keys %newflags ) {
$flags .= $_;
}
my $new_fname = "$cur_dir/$base_fname:2,$flags$extra_flags";
if ( $fname ne $new_fname ) {
rename( $fname, $new_fname )
|| print STDERR "rename($fname, $new_fname) failed: $!\n";
}
}
}
}
else {

# Dovecot to Courier keywords conversion
return unless $dovecot_mtime;

if ( !$overwrite && -f "$dir/$courier_keywords_file" ) {
print "$courier_keywords_file already exists, not overwritten\n" if ( !$quiet );
return;
}

# Read Dovecot keywords list into memory
open my $dovecot_kw_fh, '<', "$dir/$dovecot_keywords_file" || die $!;
my %keywords;
while ( my $line = readline($dovecot_kw_fh) ) {
chomp $line;
if ( $line =~ /(\d+)\s+(.+)/ ) {

# Number then Keyword
$keywords{$1} = $2;
}
}
close $dovecot_kw_fh;

# Scan files in cur for keywords
my $cur_dir = "$dir/cur";
my %file_keyword_map;

my @cur_files;
if ( opendir my $cur_dir_dh, $cur_dir ) {
@cur_files = readdir($cur_dir_dh);
closedir $cur_dir_dh;
}
foreach my $basename (@cur_files) {
my $flags;
my $extra_flags;
my $keywords = '';

# Split out and process flags
if ( $basename =~ /^([^:]+):2,([^,]*)(,.*)?$/ ) {
( $basename, $flags, $extra_flags ) = ( $1, $2, $3 );
$extra_flags = "" unless ( defined $extra_flags );
}
else {
$basename = "";
$flags = "";
$extra_flags = "";
}
foreach my $key ( sort split( //, $flags ) ) {
my $val = ord($key) - 97;
next unless ( $val >= 0 && $val < 26 );
next unless ( defined $keywords{$val} );
$keywords .= ' ' . $val;
}
if ($keywords) {
$keywords =~ s/^\s+//;
$file_keyword_map{$basename} = $keywords;
}
}

return unless ($do_conversion);

# Make courier keywords directory if necessary
my $key_dir = "$dir/$courier_keywords_dir";
unless ( -d $key_dir ) {
unlink $key_dir;
mkdir $key_dir;
chown $owner_uid, $owner_gid, $key_dir;
}

# Remove any old courier keywords files
my @courier_keywords_files;
if ( opendir my $courier_keywords_dh, $key_dir ) {
@courier_keywords_files = readdir($courier_keywords_dh);
closedir $courier_keywords_dh;
}
foreach my $file (@courier_keywords_files) {
$file = $key_dir . $file;
next unless -f $file;
unlink $file;
}

# Write courier keywords list
return unless ( scalar %keywords );
open my $courier_kw_fh, '>', "$dir/$courier_keywords_file" || die $!;
foreach my $num ( sort keys %keywords ) {
print $courier_kw_fh $keywords{$num} . "\n";
}
print $courier_kw_fh "\n";
foreach my $file ( sort keys %file_keyword_map ) {
print $courier_kw_fh $file . ':' . $file_keyword_map{$file} . "\n";
}
close $courier_kw_fh;
chown $owner_uid, $owner_gid, "$dir/$courier_keywords_file";
}
}

sub convert_subscriptions {
my ( $dir, $owner_uid, $owner_gid ) = @_;

my $courier_mtime = ( stat("$dir/$courier_subscriptions_file") )[9] || 0;
my $dovecot_mtime = ( stat("$dir/$dovecot_subscriptions_file") )[9] || 0;

# No need to convert if there is no subscriptions files
return unless ( $courier_mtime || $dovecot_mtime );

# If we're doing auto-conversion, find the newest subscription file
my $convert_to = $conversion_type;
if ( $convert_to eq 'auto' ) {
$convert_to = $dovecot_mtime > $courier_mtime ? 'courier' : 'dovecot';
}

my $src_file = "$dir/$dovecot_subscriptions_file";
my $dst_file = "$dir/$courier_subscriptions_file";
my $src_mtime = $dovecot_mtime;
my $dst_mtime = $courier_mtime;
if ( $convert_to eq 'dovecot' ) {
$src_file = "$dir/$courier_subscriptions_file";
$dst_file = "$dir/$dovecot_subscriptions_file";
$src_mtime = $courier_mtime;
$dst_mtime = $dovecot_mtime;
}

# Sanity checks..
if ( $dst_mtime && !$overwrite ) {
print "$dst_file already exists, not overwritten\n" if ( !$quiet );
return;
}
if ( $dst_mtime && !-f $dst_file ) {
print "$dst_file already exists as something other than a file\n" if ( !$quiet );
return;
}
unless ($src_mtime) {
return;
}
unless ( -f $src_file ) {
print "$src_file isn't a regular file\n" if ( !$quiet );
return;
}

return unless ($do_conversion);

open( my $src_fh, '<', $src_file ) || die $!;
open( my $dst_fh, '>', $dst_file ) || die $!;
while ( my $line = readline($src_fh) ) {
chomp $line;
if ( $line =~ /^INBOX$/i ) {
print $dst_fh "INBOX\n";
}
elsif ( $convert_to eq 'dovecot' ) {
if ( $line =~ /^INBOX\.(.*)$/i ) {
print $dst_fh "$1\n";
}
else {

# Unknown. The dovecot migrate script leaves these as-is...
print $dst_file "$line\n";
}
}
else {

# converting to Courier INBOX namespace
if ( $line =~ /\S/ ) {
print $dst_fh "INBOX.$line\n";
}
}
}
close $src_fh;
close $dst_fh;
chown $owner_uid, $owner_gid, $dst_file;
}

sub check_maildir_single {
my ( $dir, $childbox ) = @_;

$dir =~ s{^\./}{}g;

my $owner_uid;
my $owner_gid;

# Store the relevant stats()
my @courier_pop_stat = ();
@courier_pop_stat = stat("$dir/$courier_pop3_uidfile") unless $childbox;
my @courier_imap_stat = stat("$dir/$courier_imap_uidfile");
my @dovecot_stat = stat("$dir/$dovecot_uidfile");

# Gather mtimes
my $courier_pop_mtime = ( scalar @courier_pop_stat > 0 ) ? $courier_pop_stat[9] : 0;
my $courier_imap_mtime = ( scalar @courier_imap_stat > 0 ) ? $courier_imap_stat[9] : 0;
my $dovecot_mtime = ( scalar @dovecot_stat > 0 ) ? $dovecot_stat[9] : 0;

# Determine conversion type
my $convert_uidl_to = $conversion_type;

if ( $convert_uidl_to eq 'auto' ) {
$convert_uidl_to = $dovecot_mtime > $courier_pop_mtime && $dovecot_mtime > $courier_imap_mtime ? 'courier' : 'dovecot';
}

# Convert UIDLs

if ( $convert_uidl_to eq 'dovecot' ) {

# To Dovecot
unless ( $courier_pop_mtime || $courier_imap_mtime ) {
print "$dir: No imap/pop3 uidlist files\n" if ( !$quiet && !$childbox );
return;
}

$owner_uid = $courier_pop_mtime ? $courier_pop_stat[4] : $courier_imap_stat[4];
$owner_gid = $courier_pop_mtime ? $courier_pop_stat[5] : $courier_imap_stat[5];

my $uidv = -1;
my $nextuid = 1;
my $filename_map;

if ( $courier_pop_mtime) {
$filename_map = read_courier_pop3( $dir );
}

if ($courier_imap_mtime) {
( $uidv, $nextuid, $filename_map ) = read_courier_imap( $dir, $filename_map );
}
$global_mailbox_count++;
write_dovecot_uidlist( $dir, $uidv, $nextuid, $owner_uid, $owner_gid, $filename_map );
remove_dovecot_caches($dir) if ($overwrite);
}
else {

# To Courier
unless ($dovecot_mtime) {
print "$dir: No imap/pop3 uidlist files\n" if ( !$quiet && !$childbox );
return;
}

$owner_uid = $dovecot_stat[4];
$owner_gid = $dovecot_stat[5];
my ( $uidv, $nextuid, $msguids ) = read_dovecot_uidfile($dir);
if ($uidv) {
write_courier_pop3( $dir, $uidv, $nextuid, $owner_uid, $owner_gid, $msguids );
write_courier_imap( $dir, $uidv, $nextuid, $owner_uid, $owner_gid, $msguids );
}
}

# If we get here we did a UIDL conversion. Now convert subscriptions and keywords

convert_subscriptions( $dir, $owner_uid, $owner_gid );
convert_keywords( $dir, $owner_uid, $owner_gid );
}

sub remove_dovecot_caches {
my $dir = shift;
foreach my $file ( qw(dovecot.index dovecot.index.cache dovecot.index.log dovecot.index.log2), $dovecot_uidvalidity_file ) {
unlink $dir . '/' . $file;
}
unlink glob( $dir . '/' . $dovecot_uidvalidity_file . '.*' );
}

sub read_dovecot_uidfile {
my $dir = shift;
my $dovecot_uidfile = "$dir/$dovecot_uidfile";

my $uidv;
my $nextuid = 1;
my $dovecot_uid_version;
my @msguids;

if ( !-f $dovecot_uidfile ) {
print "$dovecot_uidfile: OK\n" if ( !$quiet );
return;
}

my $dovecot_uid_fh;
open( $dovecot_uid_fh, '<', $dovecot_uidfile ) || die $!;
my $dovecot_hdr = readline($dovecot_uid_fh);
if ( $dovecot_hdr =~ /^3\s+(.+)$/ ) {
my $options = $1;
$dovecot_uid_version = 3;
foreach my $part ( split( /\s+/, $options ) ) {
if ( $part =~ /(\w)(.+)/ ) {
my $type = $1;
my $val = $2;
if ( $type eq 'V' ) {
$uidv = $val;
}
elsif ( $type eq 'N' ) {
$nextuid = $val;
}
}
}

unless ($uidv) {
$global_error_count++;
print STDERR "$dovecot_uidfile: Broken header: $dovecot_hdr\n";
close $dovecot_uid_fh;
return;
}
}
elsif ( $dovecot_hdr =~ /^1\s+(\S+)\s+(\S+)$/ ) {
$dovecot_uid_version = 1;
$uidv = $1;
$nextuid = $2;
}
else {
$global_error_count++;
print STDERR "$dovecot_uidfile: Broken header: $dovecot_hdr\n";
close $dovecot_uid_fh;
return;
}

while ( my $line = readline($dovecot_uid_fh) ) {

chomp $line;
my @prts = split( /\s+/, $line );

if ( $dovecot_uid_version eq '3' ) {
next unless ( scalar @prts >= 2 );
my $msgnum = shift @prts;
my $filename = pop @prts;
$filename =~ s/^\://;
my $msgsize;

# Dovecot may or may not store the sizes for each message in the uidl file
# S# is the size with UNIX newlines, W# is the size with windows newlines
SIZE_LOOP:
foreach my $subprt (@prts) {
if ( $subprt =~ s/^W// ) {
$msgsize = $subprt;
last SIZE_LOOP;
}
}
push @msguids, [ $msgnum, $filename, $msgsize ];
$nextuid = $msgnum + 1 if ( $msgnum >= $nextuid );
}
else {

# process V1 data
next unless ( scalar @prts == 2 );
push @msguids, [ $prts[0], $prts[1], undef ];
$nextuid = $prts[0] + 1 if ( $prts[0] >= $nextuid );
}
}

return ( $uidv, $nextuid, \@msguids );
}

sub write_courier_pop3 {
my $dir = shift;
my $uidv = shift;
my $nextuid = shift;
my $owner_uid = shift;
my $owner_gid = shift;
my $msguids = shift;

# Check file/overwrite/conversion
my $uidlist_fname = "$dir/$courier_pop3_uidfile";
if ( !$overwrite && -f $uidlist_fname ) {
print "$uidlist_fname already exists, not overwritten\n" if ( !$quiet );
return;
}

return if ( !$do_conversion );

# Check that all entries have sizes.. We don't write the pop3 file if we can't do so correctly
foreach my $msg_ar ( @{$msguids} ) {
unless ( defined $msg_ar->[2] ) {
print "Missing some sizes for $uidlist_fname, skipping\n" if ( !$quiet );
return;
}
}

# Write file
open( my $courier_pop_fh, '>', $uidlist_fname ) || die $!;
print $courier_pop_fh "/2 $nextuid $uidv\n";
foreach my $msg_ar ( @{$msguids} ) {
print $courier_pop_fh $msg_ar->[1] . ' ' . $msg_ar->[2] . ' ' . $msg_ar->[0] . ':' . $uidv . "\n";
}
close $courier_pop_fh;
chown $owner_uid, $owner_gid, $uidlist_fname;
$uidlist_write_count++;

}

sub write_courier_imap {
my $dir = shift;
my $uidv = shift;
my $nextuid = shift;
my $owner_uid = shift;
my $owner_gid = shift;
my $msguids = shift;

# Check file/overwrite/conversion
my $uidlist_fname = "$dir/$courier_imap_uidfile";
if ( !$overwrite && -f $uidlist_fname ) {
print "$uidlist_fname already exists, not overwritten\n" if ( !$quiet );
return;
}

return if ( !$do_conversion );

# Write file
open( my $courier_imap_fh, '>', $uidlist_fname ) || die $!;
print $courier_imap_fh "1 $uidv $nextuid\n";
foreach my $msg_ar ( @{$msguids} ) {
my $filename = $msg_ar->[1];
$filename =~ s/\:2.*$//;
print $courier_imap_fh $msg_ar->[0] . ' ' . $filename . "\n";
}
close $courier_imap_fh;
chown $owner_uid, $owner_gid, $uidlist_fname;
$uidlist_write_count++;

}

sub check_maildir {
my ($dir) = @_;

my $orig_mailboxes = $global_mailbox_count;

check_maildir_single( $dir, 0 );
my @check_maildir_files;
if ( opendir my $check_maildir_dh, $dir ) {
@check_maildir_files = readdir($check_maildir_dh);
closedir $check_maildir_dh;
}
foreach my $file (@check_maildir_files) {
next unless ( $file =~ /^\./ );
next if ( $file =~ /^\.?\.$/ );
$file = $dir . '/' . $file;
next if ( -l $file );
check_maildir_single( $file, 1 );
}

$global_user_count++ if ( $orig_mailboxes != $global_mailbox_count );
}

sub is_maildir {
my ($dir) = @_;

# Do we need to check for the courier specific files here or is it enough to assume every maildir will have a cur directory?
return ( -f "$dir/$courier_pop3_uidfile" || -f "$dir/$courier_imap_uidfile" || -d "$dir/cur" );
}

sub userdir_check {
my ($dir) = @_;
my $found = 0;

my @userdir_check_files;
if ( opendir my $userdir_dh, $dir ) {
@userdir_check_files = readdir($userdir_dh);
closedir $userdir_dh;
}
foreach my $userdir (@userdir_check_files) {
$userdir = $dir . '/' . $userdir;
next if ( -l $userdir );
next if ( !-d $userdir );

if ( $maildir_subdirs == -1 ) {

# unknown if we want $maildir_name/ or not
if ( -d "$userdir/$maildir_name" && is_maildir("$userdir/$maildir_name") ) {
$maildir_subdirs = 1;
}
elsif ( is_maildir($userdir) ) {
$maildir_subdirs = 0;
}
else {
next;
}
}

if ( $maildir_subdirs == 1 ) {
if ( is_maildir("$userdir/$maildir_name") ) {
check_maildir("$userdir/$maildir_name");
$found = 1;
}
}
elsif ( $maildir_subdirs == 0 ) {
if ( is_maildir($userdir) ) {
check_maildir($userdir);
$found = 1;
}
}
}
return $found;
}

sub depth_check {
my ( $dir, $depth ) = @_;
my $found = 0;

my @depth_check_files;
if ( opendir my $depth_check_dh, $dir ) {
@depth_check_files = readdir($depth_check_dh);
closedir $depth_check_dh;
}
foreach my $subdir (@depth_check_files) {
next if ($subdir eq '.' || $subdir eq '..');
$subdir = $dir . '/' . $subdir;
next if ( !-d $subdir );

if ( $depth > 0 ) {
$found = 1 if ( depth_check( $subdir, $depth - 1 ) );
}
else {
$found = 1 if ( userdir_check($subdir) );
}
}
return $found;
}

sub usage {
print "Usage: maildir-migrate [options] <maildir>\n\n";
print "Options:\n";
print " --convert Perform conversion\n";
print " --quiet Silence output\n";
print " --overwrite Overwrite existing files\n";
print " --recursive Recursively look through maildir for subaccounts\n";
print " --to-dovecot Conversion is from Courier to Dovecot\n";
print " --to-courier Conversion is from Dovecot to Courier\n";
exit 0;
}
</pre>

Dovecot

2010-03-06T12:11:15+00:00

← Older revision		Revision as of 12:11, 6 March 2010
Line 52:		Line 52:
	* [[GMail]]		* [[GMail]]
	* [[My Email Services Configurations]]		* [[My Email Services Configurations]]
		+	* [[Converting from Courier IMAP to Dovecot IMAP]]

	== External Pages ==		== External Pages ==

Perdition 8 Manual Page

2010-03-06T10:21:51+00:00

Created page with '<pre> Perdition: perdition(8) Manual Page Section: Maintenance Commands (8) Updated: 12th June 2003 Index Return to Main Contents NAME perdition - POP3 and IMAP4 proxy server ...'

New page

<pre>
Perdition: perdition(8) Manual Page

Section: Maintenance Commands (8)
Updated: 12th June 2003
Index Return to Main Contents
NAME
perdition - POP3 and IMAP4 proxy server
SYNOPSIS
perdition [options]
perdition.pop3 [options]
perdition.pop3s [options]
perdition.imap4 [options]
perdition.imap4s [options]
perdition.imaps [options]
DESCRIPTION
perdition allows users to connect to a content-free POP3 or IMAP4 server that will redirect them to their real POP3 or IMAP4 server respectively. This enables mail retrieval for a domain to be split across multiple real mail servers on a per user basis. This can also be used to as a POP3 or IMAP4 proxy especially in firewall applications.

When a connection is made to perdition in POP3 mode, it reads the USER and PASS commands and then refers to its popmap to find the real-server that the user's connection should be forwarded to. A connection is made to the real-server and perdition then enters the USER and PASS commands. If authentication is successful then perdition pipes data between the end-user and the real-server. If authentication fails then the real-server connection is closed and the client connection is reset to the state it was in on initial connection. That is new USER and PASS commands are expected. Similarly in IMAP4 mode, perdition accepts the LOGIN command and passes the username and password onto the real-server or authentication.

No IMAP authentication schemes, other than the LOGIN command are accepted.

When invoked as perdition.pop3, perdition.pop3s, perdition.imap4 or perdition.imap4s then perdition will run in POP3, POP3S, IMAP4 or IMAP4S mode respectively, unless overridden on the command line or in the configuration file. perdition.imaps also runs perdition in IMAP4S mode and is provided to get around the truncation of process names in the /proc filesystem on Linux which can cause init scripts to fail to stop perdition correctly.
OPTIONS

-A|--add_domain STATE[,STATE...][,STRIP_DEPTH]:
Appends a domain to the USER based on the IP address connected to in given state(s). The domain name to append will be the reverse-lookup of the IP address connected to. If there is no reverse lookup for this IP address, then a domain will not be appended. Probably the easiest way to enforce this mapping is to add entries to /etc/hosts.

The valid states are servername_lookup, local_authentication, remote_login and all

servername_lookup: Append the domain to the username for lookup of username in Popmap. Will not take effect if client_server_specification is in effect.

local_authentication: Append the domain to the username for use in local authentication. Only has effect if authenticate_in is in effect.

remote_login: Send the username with the domain appended to the real-server for authentication.

all: Short-Hand for all of above states.

The domain may also have leading levels striped, essentially to convert a hostname to a domain name. The depth of the strip defaults to 1, which would mean that www.au.vergenet.net would become au.vergenet.net. A depth of 2 would cause it to become vergenet.net and so forth. A depth of 0 leaves the name unchanged. The depth and may be specified by appending ",STRIP_DEPTH" to the state. For compatibility reasons the default depth is 1.

e.g. all,2

(the default value for add_domain is "")
-a, --authenticate_in:
User is authenticated by perdition before connection to back-end server is made. Only available if perdition is compiled with pam support.
-B, --no_bind_banner:
If bind_address is specified, then the address will be resolved and the reverse-lookup of this will be used in the greeting. This option disables this behaviour an reverts to using the uname to derive the hostname for the greeting.
-b, --bind_address SERVER[,SERVER...]:
Bind to these addresses and ports. interfaces with this address. Format is as per the --outgoing_server option. If the port is ommitted, then the listen_port will be used.

In non-inetd mode, connections will only be accepted to the listed servers. If un-set connections will be accepted on all addresses on the listen_port.

(default "")
-C|--connection_logging:
Log interaction between clients, perdition and servers during authentication phase.

Note: -d|--debug must be specified for this option to take effect.
--connect_relog SECONDS:
How often to relog the connection. For use in conjunction with POP and IMAP before SMTP. If zero then the connection will not be reloged.
(default 300)
-c, --client_server_specification:
Allow USER of the form user<delimiter>server[:port] to specify the server and port for a user.
-D, --domain_delimiter STRING:
Delimiter between username and domain.
(default "@")
-d, --debug:
Turn on verbose debugging.
-e, --explicit_domain STRING:
With -A, use STRING as the default domain rather than deriving from the IP address connected to.
(default NULL)
-F, --log_facility FACILITY:
Facility to log to. If the facility has a leading '/' then it will be treated as a file. If is "-" or "+" then log to stdout or stderr respectively. Otherwise it is assumed to be the name of a syslog facility. See syslog.conf(5) for valid syslog facility names.
(default "mail")
Notes: If an error occurs before options are read it may be logged to stderr. If stdout or stderr is specified as the facility, then the process will not fork and detach from the terminal.
-f, --config_file FILENAME:
Name of config file to read. Command line options override options set in config file.

The default is derived as follows:

The sysconfig dir ("/etc/perdition" for example) is checked for <basename>.conf. If this is found then it is used. So if perdition is invoked as /usr/sbin/perdition.pop3, and /etc/perdition/perdition.pop3.conf exists then it will be used.

Next the sysconfig dir is checked for peridtion.<protocol>.conf, where protocol is the ASCII representation of the protocol being used, one of "imap4", "imap4s", "pop3", or "pop3s". So if perdition is being run in imap4 mode, and /etc/perdition/perdition.imap4.conf exists, then it is used. Note that the protocol name is lowercase.

Next the sysconfig dir is checked for perdition.conf, if it is found then it is used.

If none of these files are found then no configuration file is used.
-g, --group GROUP:
Group to run as.
(default "nobody")
-h, --help:
Display this message
-I, --capability, --pop_capability, --imap_capability STRING:
Capabilities for the protocol.

When using a POP3 based protocol, the capabilities should be delimited by two spaces. This is because the capabilities themselves may contain single spaces. The default is "UIDL USER".

When using an IMAP4 based protocol, this string is taken as a string literal that will be returned when a client issues the CAPABILITY command. As such the capabilities should be space delimited. The default is "IMAP4 IMAP4REV1". However, perdition does support RFC 2088 non-synchronising string literals, if the real servers also support this then the capability may be set to "IMAP4 IMAP4REV1 LITERAL+".

If perdition is listening for TLS connections then the capability STLS for POP3 or STARTTLS for IMAP4 will be appended to the list of capabilities if it is not already present. Similarly these capabilities will be removed from the list of capabilities if they are present and perdition is not listening for TLS connections.

Perdition may also manipulate the capability in IMAP mode to add and remove the LOGINDISABLED capability if the no_login capability is in effect or if the ssl_mode includes tls_listen_force or tls_outgoing_force.
-i, --inetd_mode:
Run in inetd mode
-L, --connection_limit LIMIT:
Maximum number of connections to accept simultaneously. A value of zero sets no limit on the number of simultaneous connections.
(default 0)
-l, --listen_port PORT_NUMBER|PORT_NAME:
Port to listen on.

The default is 110, 995, 143 and 993 in POP3, POP3S, IMAP4 and IMAP4S mode respectively.
--login_disabled:
Do not allow users to log in. Also adds LOGINDISABLED to capability list in IMAP4 and IMAP4S mode.
--log_passwd STATE:
Log the users password, otherwise just report it as "XXXXXX".
(default "never")

fail: log the password on failed connection attepmts.

ok: log the password on successful connection attepmts.

never: never log the password

always: always log the password
--lower_case state[,state...]:
Convert usernames to lower case according the the locale in given state(s). See A|add_domain for a description of the states.
(default "(null)")
-M, --map_library FILENAME:
Library to open that provides functions to look up the server for a user. An empty ("") library means that no library will be accessed and hence, no lookup will take place.
(default "/usr/lib/libperditiondb_gdbm.so.0")
-m, --map_library_opt STRING:
String option to pass to database access function provided by the library specified by the map_library directive. The treatment of this string is up to the library. See perditiondb(5) for more details of how individual map_libraries handle this string.
(default "")
--no_daemon:
Do not detach from terminal. Makes no sense if inetd_mode is in effect.
-n, --no_lookup:
Disable host and port lookup, implies no_bind_banner. Please note that if this option is enabled, then perdition will not resolve host or port names returned by popmap lookups, thus, your popmap must return ip addresses and port numbers.
-O, --ok_line:
Use STRING as the OK line to send to the client. Overridden by server_resp_line. OK and will be prepended to STRING, and in IMAP mode a tag will also be prepended to the string.
(default "You are so in")
--server_ok_line:
This option is deprecated and may be removed in a future release. Use server_resp_line instead. If authentication with the real-server is successful then send the servers +OK line to the client, instead of generating one.
-o, --server_resp_line:
If authentication with the real-server is successful then send the servers response line to the client, instead of generating one.
-P, --protocol PROTOCOL:
Protocol to use.
(default "POP3") available protocols: "POP3, POP3S, IMAP4, IMAP4S"
-p, --outgoing_port PORT:
Default real-server port.
See listen_port for defaults.
-s, --outgoing_server SERVER[,SERVER...]:
Define a server to use if a user is not in the popmap. Format is servername|ip_address[:portname|portnumber]. Multiple servers may be delimited by a ','. If multiple servers are specified then they are used in a round robin fashion.
(default "")
--pid_file FILENAME:
Path for pidfile. Must be a full path starting with a '/'. To allow perdition to remove the pid file after the owner of the perdition process is changed to a non-root user, it is advised to specify a pid file in a subdirectory of the system var state directory (usually /var/run). This subdirectory should be unique to this perdition invocation and will be created and have its owner and permissions set to allow perdition to subsequently removed the pid file.
Empty for no pid file. Not used in inetd mode.
(default <var_state_dir>/<basename>/<basename>.pid)
-S, --strip_domain STATE[,STATE]:
Allow USER of the from user<delimiter>domain where <delimiter>domain will be striped off in given state(s).See add_domain for a description of the states.
-t, --timeout SECONDS:
Idle timeout. Zero for infinite timeout.
(default 1800)
-u, --username USERNAME:
User to run as.
(default "nobody")
-U, --username_from_database:
If the servername in the popmap specified in the form: user<delimiter>domain then use the username given by the servername. If a servername is given in this form then the domain will be used as the server to connect to, regardless of this option.
-q, --quiet:
Only log errors. Overridden by debug
--query_key FORMAT[,FORMAT...]:
Instead of using the username as supplied by the end user, possibly modified by strip_domain, use the formats specified. The formats will be used in order to query the popmap. The result from the first successful lookup will be used. The format is comprised of a string of characters, delimited by ','. The following escape codes are valid:

\U: Long Username, the entire string supplied by
the end user, less any effects of
--strip_domain.
\u: Short Username, the portion Long Username
before the domain delimiter.
\D: Domain Delimiter, as specified by
--domain_delimiter
\d: Domain the portion Long Username after the
domain delimiter.
\i: Source IP address of the connection
\I: Destination IP address of the connection
\p: Source port of the connection
\P: Destination port of the connection
\\: Literal \

As a ',' is the delimiter between formats, it cannot appear within a format. All other characters other than the escape codes above, and ',' are treated as literals.

Examples

Use the supplied username, the default behaviour
\U

Use the user portion of the supplied username, if this doesn't work try the domain portion of the supplied username preceded by the domain delimiter
\u,\D\d

Use the destination IP address
\I

Escape codes interspersed with literals
\u\da_domain,\da_domain

The options below relate to SSL/TLS support. They are not available if perdition is compiled without SSL support.

--ssl_mode MODE:
Use SSL and or TLS for the listening and/or outgoing connections. A comma delimited list of: none, ssl_listen, ssl_outgoing, ssl_all, tls_listen, tls_outgoing, tls_all, tls_listen_force, tls_outgoing_force, tls_all_force. TLS is defined in RFC 2595.
(default "(null)")

none: Do not use SSL or TLS for any connections. This is the same as providing no option, the default.

ssl_listen: When listening for incoming connections they will be treated as SSL connections.

ssl_outgoing: Use SSL to connect to real pop/imap servers.

ssl_all: Short-Hand for ssl_listen,ssl_outgoing.

tls_listen: When listening for incoming connections they will be treated as TLS connections.

tls_outgoing: Use TLS to connect to real pop/imap servers.

tls_all: Short-Hand for tls_listen,tls_outgoing.

tls_listen_force: Do not accept plain text authentication. In IMAP4 and IMAP4S mode, the LOGINDISABLED capability until TLS has been initialised by the client issuing a STARTTLS. In all modes mode plain-text authentication is ignored. Also sets tls_listen.

tls_outgoing_force: Do not send authentication information if TLS cannot be negotiated. Also sets tls_outgoing.

tls_all_force: Short-Hand for tls_listen_force,tls_outgoing_force.
--ssl_ca_chain_file:
Sets the optional all-in-one file where you can assemble the certificates of Certification Authorities (CA) which form the certificate chain of the server certificate. This starts with the issuing CA certificate of the "ssl_cert_file" certificate and can range up to the root CA certificate. Such a file is simply the concatenation of the various PEM-encoded CA Certificate files, usually in certificate chain order. Overrides ssl_ca_file and ssl_ca_path.
(default NULL, no CA certificate will be used)
--ssl_ca_file FILENAME:
Certificate Authorities to use when verifying certificates of real servers. Used for SSL or TLS outgoing connections. When building the Certificate Authorities chain, ssl_ca_file is used first, if set, and then ssl_ca_path, if set. See SSL_CTX_load_verify_locations(3) for format details.
(default "/etc/perdition/perdition.ca.pem")
--ssl_ca_path PATHNAME:
Certificate Authorities to use when verifying certificates of real servers. Used for SSL or TLS outgoing connections. "openssh c_rehash" should be run in this directory when new certificates are added. When building the Certificate Authorities chain, ssl_ca_file is used first, if set, and then ssl_ca_path, if set. See SSL_CTX_load_verify_locations(3) for details.
(default "/etc/perdition/perdition.ca/")
--ssl_ca_accept_self_signed:
Accept self-signed certificate authorities.
--ssl_cert_file FILENAME:
Certificate to use when listening for SSL or TLS connections. Should be in PEM format.
(default "/etc/perdition/perdition.crt.pem")
--ssl_cert_accept_self_signed:
Accept self-signed certificates. Used for SSL or TLS outgoing connections.
--ssl_cert_accept_expired:
Accept expired certificates. This includes server certificates and certificate authority certificates. Used for SSL or TLS outgoing connections.
--ssl_cert_accept_not_yet_valid:
Accept certificates that are not yet valid. This includes server certificates and certificate authority certificates. Used for SSL or TLS outgoing connections.
--ssl_cert_verify_depth DEPTH:
Chain Depth to recurse to when verifying certificates. Used for SSL or TLS outgoing connections.
(default 9)
--ssl_key_file FILENAME:
Public key to use when listening for SSL or TLS connections. Should be in PEM format.
(default "/etc/perdition/perdition.key.pem")
--ssl_listen_ciphers STRING:
Cipher list when listening for SSL or TLS connections as per ciphers(1). If empty ("") then openssl's default will be used.
(default "")
--ssl_outgoing_ciphers STRING:
Cipher list when making outgoing SSL or TLS connections as per ciphers(1). If empty ("") then openssl's default will be used.
(default "")
--ssl_no_cert_verify:
Don't cryptographically verify the real-server's certificate. Used for SSL or TLS outgoing connections.
--ssl_no_cn_verify:
Don't verify the real-server's common name with the name used. to connect to the server. Used for SSL or TLS outgoing connections.
Notes:
Default value for binary flags is off.
If a string argument is empty ("") then the option will not be used unless noted otherwise.
The defaults given refer to the values if perdition is compiled with --sysconfdir=/etc as it would for many binary distributions. For the actual defaults of a given perdition binary run "perdition --help"

USER DATABASE (POPMAP)
For information on mechanisms for resolving users to a host and port and information on the -M|--map_library and -m|--map_library_opt flags, please see perditiondb(5).

Note that by specifying an map library no map lookups will occur and all connections will use the -s|--outgoing_server. In this way perdition can be configured as a "pure proxy".
STAND-ALONE MODE
Normally perdition will bind to a port, and listen for connections. The default port is 110 in POP3 mode and 143 in IMAP4 mode, an alternate port can be specified with the -l|--listen_port command line option. In this mode perdition will fork to manage clients.

Stand-Alone Mode: Debian and RPM Installation

In the Debian and RPM distributions perdition can be started and stopped in stand-alone mode using:

/etc/init.d/perdition start
/etc/init.d/perdition stop

Editing /etc/sysconfig/perdition (RPM) or /etc/default/perdition (Debian) allows control of whether perdition will be started in POP3 mode, IMAP4 mode or both (or neither).

The syntax for this file is:

RUN_PERDITION=[yes|no]
FLAGS="flags"
POP3=[yes|no]
POP3_FLAGS="flags"
POP3S=[yes|no]
POP3S_FLAGS="flags"
IMAP4=[yes|no]
IMAP4_FLAGS="flags"
IMAP4S=[yes|no]
IMAP4S_FLAGS="flags"

The file is sourced into the init script so normal bash syntax applies. Blank lines are ignored, as is anything after a # on a line.

e.g.

RUN_PERDITION=yes
POP3=on
POP3_FLAGS="--ssl_mode tls_listen"
POP3S=on
IMAP4_FLAGS="--ssl_mode tls_listen"
IMAP4=on
POP3S_FLAGS="--ssl_mode ssl_listen -p 110"
IMAP4S=on
IMAP4S_FLAGS="--ssl_mode ssl_listen -p 143"

INETD MODE
Perdition can be used in conjunction with inetd. This enables perdition to benefit from tcpd where access can be controlled to some extent using /etc/hosts.allow and /etc/hosts.deny. Sample /etc/inetd.conf entries follow:

pop3 stream tcp nowait root /usr/sbin/tcpd /usr/sbin/perdition.pop3 -i
pop3s stream tcp nowait root /usr/sbin/tcpd /usr/sbin/perdition.pop3s -i
imap2 stream tcp nowait root /usr/sbin/tcpd /usr/sbin/perdition.imap4 -i
imaps stream tcp nowait root /usr/sbin/tcpd /usr/sbin/perdition.imap4s -i

inetd should then be restarted
LOCAL AUTHENTICATION
If perdition has been compiled against libpam, it may be set up to authenticate the user locally once the USER and PASS commands are entered by specifying the -a|--authenticate_in option on the command line. This authentication happens before the connection to the foreign server is made and must succeed for a connection to the foreign server to be made.

This authentication uses PAM and a sample pam configuration file for perdition can be found in etc/pam.d/perdition in the source tree. This should be dropped into /etc/pam.d.
DOMAIN DELIMITER
A multi character domain delimiter can be set using the -d|--domain delimiter option. This sets the delimiter used in conjunction with the -S|--strip_domain and -c|--client_server_specification options.
USER PORT SPECIFICATION
If perdition is invoked with the -c|--client_server_specification flag then the user may optionally specify the server and port that perdition should connect to for the client using the syntax user<delimiter>host[:port].

Example:

IMAP4

0 login henry@that.host:143

POP3

user james@other.host

IDLE TIMEOUTS
If there is no input from the client or back-end server for greater than timeout seconds then the connection is closed. The default timeout is 1800 seconds and can be specified on the command line with the -t|--timeout option. A time out of 0 means that timeouts are disabled and clients and back-end servers can idle indefinitely.
LOOP DETECTION
The greeting that perdition displays when accepting an incoming connection is "+OK POP3 Ready <hostname>" or "* OK IMAP4 Ready <hostname>" in POP3 and IMAP4 modes respectively. If when perdition connects to the back-end server the greeting string matches the greeting string of the perdition process making the connection then it is assumed that perdition is connecting to itself and a "Re-Authentication Failure" is returned to the client.
CONFIGURATION FILE
The format of a line of the configuration file is:

<key> <value>

Key is either a short or long option as per perdition -h|--help, without the leading - or --. Blank lines are ignored, as is anything including and after a # (hash) on a line. If a \ precedes a new line then the lines will be concatenated. IF a \ precedes any other character, including a # (hash) it will be treated as a literal. Anything inside single quotes (') will be treated as a literal. Anything other than a (') inside double quotes (") will be treated as a literal. Whitespace in keys must be escaped or quoted. Whitespace in values need not be escaped or quoted.

Options that do not make sense in the configuration file such as h|help and f|config_file are ignored. Options specified on the command line override the options in this file.

Example configuration File.

# perdition.conf
l 110 #Short option used as key
group mail #Long option used as key
a #Option with no argument

POP BEFORE SMTP
Perdition supports POP before SMTP in both POP3 and IMAP4 mode by logging having logging the following messages:

When a user connects:

Connect: <source_ip_address>[inetd_pid=<pid>]

When a user is authenticated

Auth: <source_ip_address> user="<username>" passwprd="<passwowd>" server="<servername"> port="<port>" status=failed...|ok

When a user disconnects

Close: <source_ip_address> user="<username>" received=<bytes> sent=<bytes>
LOGGING
By default, logs are logged via syslog using the facility mail. You should inspect /etc/syslog.conf to find where these logs are written. Under Debian these logs will be written to /var/log/mail.log, under Red Hat 7.x these logs will be written to /var/log/maillog, under Solaris 8 these logs will be written to /var/log/syslog. Normally each session will have two perdition log entries. Logs are prepended, depending on syslog with the date, host, and perdition[<pid>]: .

Fatal errors are also logged with a priority of err. In stand-alone mode the startup parameters are logged on initialisation. If the -d|--debug command line option or configuration file directive is used then startup parameters are logged regardless of other configuration directives and in both stand-alone and identd mode additional debugging messages are logged with a priority of debug. As the flag implies, this is useful for debugging but is probably too verbose for production systems. If the -q|--quiet command line option or configuration file directive is used, only errors will be logged. This is overridden by -d|--debug.
SSL/TLS Support
Perdition supports using SSLv2 and SSLv3 to encrypt sessions between end users and perdition and sessions between perdition and real servers. SSL may be used for either, both or none of these classes of connections.

The public key and certificate files should be in PEM format. As a quick guide, the files may be generated using openssl with the following command:

openssl req -new -x509 -nodes \
-out perdition.crt.pem -keyout perdition.key.pem -days 365

FILES

/etc/perdition/perdition.conf

SEE ALSO
perditiondb(5), inetd(8), syslog.conf(5), syslogd(8)
AUTHORS
Lead
Horms

Perditiondb Library Authors
Frederic Delchambre (MySQL)
Chris Stratford: (LDAP and Berkely DB)
Nathan Neulinger (NIS)

Contributing Authors
Daniel Roesen
Clinton Work
Youri
Jeremy Nelson
Wim Bonis
Arvid Requate
Mikolaj J. Habryn
Ronny Cook
Geoff Mitchell
Willi Langenberger
Matt Prigge
Wolfgang Breyha

Index

NAME
SYNOPSIS
DESCRIPTION
OPTIONS
USER DATABASE (POPMAP)
STAND-ALONE MODE
INETD MODE
LOCAL AUTHENTICATION
DOMAIN DELIMITER
USER PORT SPECIFICATION
IDLE TIMEOUTS
LOOP DETECTION
CONFIGURATION FILE
POP BEFORE SMTP
LOGGING
SSL/TLS Support
FILES
SEE ALSO
AUTHORS
</pre>

== Related Pages ==
* [[Perdition]]

Perdition(8) Manual Page

2010-03-06T10:21:35+00:00

Redirected page to Perdition 8 Manual Page

Show changes

Redis, MemcacheDB, or CouchDB?

2010-03-06T08:11:58+00:00

I’m having a great time exploring lots of cool Ruby stuff and now I’m eager to get into the document-centric (non SQL) databases that are currently the buzz.

These are also known as simple key-value pair databases; their use makes them different now. They are now on the network, and the “value” they store are full documents.

There are three that I know of, though I’m sure there are more:

Redis
CouchDB
MemcacheDB

Redis is new to the scene, but its been getting rave reviews.

CouchDB is from the awesome folks at the Apache Foundation and can likely support traffic the likes of Facebook, hence more than I need. ~~Its Java based and for me that’s a drawback.~~ Its based on Erlang, which is cool with me. Thanks grant!

MemcacheDB is right up my alley. (I’m actually already using it, though for some fairly simple reasons; not for full “documents”.) Its based on the rock solid and simple as can be memcache protocol from the folks at Danga. Its been around for a few years and I see that as a very good thing.

While my curiosity is pulling me towards Redis and CouchDB, I’m going to stick with MemcacheDB for now.

UPDATE: Shortly after posting this, I quickly found a couple more:

MongoDB
Tokyo Cabinet

Even on top of all these, I’m sure there are even more!

Squid 3 Cache Manager CGI

2010-03-06T01:43:57+00:00

Created page with '<pre> sample_start_time = 1267835856.857247 (Sat, 06 Mar 2010 00:37:36 GMT) sample_end_time = 1267839457.456888 (Sat, 06 Mar 2010 01:37:37 GMT) client_http.requests = 0.444926/se...'

New page

<pre>
sample_start_time = 1267835856.857247 (Sat, 06 Mar 2010 00:37:36 GMT)
sample_end_time = 1267839457.456888 (Sat, 06 Mar 2010 01:37:37 GMT)
client_http.requests = 0.444926/sec
client_http.hits = 0.276065/sec
client_http.errors = 0.000000/sec
client_http.kbytes_in = 0.356607/sec
client_http.kbytes_out = 1.309504/sec
client_http.all_median_svc_time = 0.013867 seconds
client_http.miss_median_svc_time = 0.127833 seconds
client_http.nm_median_svc_time = 0.000000 seconds
client_http.nh_median_svc_time = 0.007665 seconds
client_http.hit_median_svc_time = 0.000000 seconds
server.all.requests = 0.442704/sec
server.all.errors = 0.000000/sec
server.all.kbytes_in = 1.283675/sec
server.all.kbytes_out = 0.333833/sec
server.http.requests = 0.441871/sec
server.http.errors = 0.000000/sec
server.http.kbytes_in = 1.278954/sec
server.http.kbytes_out = 0.333278/sec
server.ftp.requests = 0.000000/sec
server.ftp.errors = 0.000000/sec
server.ftp.kbytes_in = 0.000000/sec
server.ftp.kbytes_out = 0.000000/sec
server.other.requests = 0.000833/sec
server.other.errors = 0.000000/sec
server.other.kbytes_in = 0.004721/sec
server.other.kbytes_out = 0.000555/sec
icp.pkts_sent = 0.446037/sec
icp.pkts_recv = 0.446037/sec
icp.queries_sent = 0.000278/sec
icp.replies_sent = 0.445759/sec
icp.queries_recv = 0.445759/sec
icp.replies_recv = 0.000278/sec
icp.replies_queued = 0.000000/sec
icp.query_timeouts = 0.000278/sec
icp.kbytes_sent = 0.055824/sec
icp.kbytes_recv = 0.057490/sec
icp.q_kbytes_sent = 0.000000/sec
icp.r_kbytes_sent = 0.055824/sec
icp.q_kbytes_recv = 0.057768/sec
icp.r_kbytes_recv = 0.000000/sec
icp.query_median_svc_time = 0.000000 seconds
icp.reply_median_svc_time = 0.000000 seconds
dns.median_svc_time = 0.046388 seconds
unlink.requests = 0.006388/sec
page_faults = 0.000555/sec
select_loops = 110.426607/sec
select_fds = 4.224019/sec
average_select_fd_period = 0.000000/fd
median_select_fds = 0.000000
swap.outs = 0.068877/sec
swap.ins = 1.070377/sec
swap.files_cleaned = 0.000000/sec
aborted_requests = 0.000000/sec
syscalls.disk.opens = 0.614064/sec
syscalls.disk.closes = 0.614064/sec
syscalls.disk.reads = 0.535189/sec
syscalls.disk.writes = 0.298284/sec
syscalls.disk.seeks = 0.000000/sec
syscalls.disk.unlinks = 0.006388/sec
syscalls.sock.accepts = 0.867356/sec
syscalls.sock.sockets = 0.374938/sec
syscalls.sock.connects = 0.373827/sec
syscalls.sock.binds = 0.374938/sec
syscalls.sock.closes = 0.820974/sec
syscalls.sock.reads = 1.607232/sec
syscalls.sock.writes = 1.421708/sec
syscalls.sock.recvfroms = 0.867911/sec
syscalls.sock.sendtos = 0.468811/sec
cpu_time = 9.062743 seconds
wall_time = 3600.599641 seconds
cpu_usage = 0.251701%
</pre>

[[Category: Squid]]

Squid

2010-03-06T01:43:42+00:00

← Older revision		Revision as of 01:43, 6 March 2010
Line 8:		Line 8:
	</pre>		</pre>

		+	== More Squid Info ==
		+	* [[Squid 3 Cache Manager CGI]]
	* [[Squid 3 To Block HTTP Headers]]		* [[Squid 3 To Block HTTP Headers]]

MySQL Stop Words

2010-03-06T01:36:13+00:00

Created page with '<pre> a's, able, about, above, according, accordingly, across, actually, after, afterwards, again, against, ain't, all, allow, allows, almost, alone, along, already, also, althou...'

New page

<pre>
a's, able, about, above, according, accordingly, across, actually, after, afterwards, again, against, ain't, all, allow, allows, almost, alone, along, already, also, although, always, am, among, amongst, an, and, another, any, anybody, anyhow, anyone, anything, anyway, anyways, anywhere, apart, appear, appreciate, appropriate, are, aren't, around, as, aside, ask, asking, associated, at, available, away, awfully, be, became, because, become, becomes, becoming, been, before, beforehand, behind, being, believe, below, beside, besides, best, better, between, beyond, both, brief, but, by, c'mon, c's, came, can, can't, cannot, cant, cause, causes, certain, certainly, changes, clearly, co, com, come, comes, concerning, consequently, consider, considering, contain, containing, contains, corresponding, could, couldn't, course, currently, definitely, described, despite, did, didn't, different, do, does, doesn't, doing, don't, done, down, downwards, during, each, edu, eg, eight, either, else, elsewhere, enough, entirely, especially, et, etc, even, ever, every, everybody, everyone, everything, everywhere, ex, exactly, example, except, far, few, fifth, first, five, followed, following, follows, for, former, formerly, forth, four, from, further, furthermore, get, gets, getting, given, gives, go, goes, going, gone, got, gotten, greetings, had, hadn't, happens, hardly, has, hasn't, have, haven't, having, he, he's, hello, help, hence, her, here, here's, hereafter, hereby, herein, hereupon, hers, herself, hi, him, himself, his, hither, hopefully, how, howbeit, however, i'd, i'll, i'm, i've, ie, if, ignored, immediate, in, inasmuch, inc, indeed, indicate, indicated, indicates, inner, insofar, instead, into, inward, is, isn't, it, it'd, it'll, it's, its, itself, just, keep, keeps, kept, know, knows, known, last, lately, later, latter, latterly, least, less, lest, let, let's, like, liked, likely, little, look, looking, looks, ltd, mainly, many, may, maybe, me, mean, meanwhile, merely, might, more, moreover, most, mostly, much, must, my, myself, name, namely, nd, near, nearly, necessary, need, needs, neither, never, nevertheless, new, next, nine, no, nobody, non, none, noone, nor, normally, not, nothing, novel, now, nowhere, obviously, of, off, often, oh, ok, okay, old, on, once, one, ones, only, onto, or, other, others, otherwise, ought, our, ours, ourselves, out, outside, over, overall, own, particular, particularly, per, perhaps, placed, please, plus, possible, presumably, probably, provides, que, quite, qv, rather, rd, re, really, reasonably, regarding, regardless, regards, relatively, respectively, right, said, same, saw, say, saying, says, second, secondly, see, seeing, seem, seemed, seeming, seems, seen, self, selves, sensible, sent, serious, seriously, seven, several, shall, she, should, shouldn't, since, six, so, some, somebody, somehow, someone, something, sometime, sometimes, somewhat, somewhere, soon, sorry, specified, specify, specifying, still, sub, such, sup, sure, t's, take, taken, tell, tends, th, than, thank, thanks, thanx, that, that's, thats, the, their, theirs, them, themselves, then, thence, there, there's, thereafter, thereby, therefore, therein, theres, thereupon, these, they, they'd, they'll, they're, they've, think, third, this, thorough, thoroughly, those, though, three, through, throughout, thru, thus, to, together, too, took, toward, towards, tried, tries, truly, try, trying, twice, two, un, under, unfortunately, unless, unlikely, until, unto, up, upon, us, use, used, useful, uses, using, usually, value, various, very, via, viz, vs, want, wants, was, wasn't, way, we, we'd, we'll, we're, we've, welcome, well, went, were, weren't, what, what's, whatever, when, whence, whenever, where, where's, whereafter, whereas, whereby, wherein, whereupon, wherever, whether, which, while, whither, who, who's, whoever, whole, whom, whose, why, will, willing, wish, with, within, without, won't, wonder, would, would, wouldn't, yes, yet, you, you'd, you'll, you're, you've, your, yours, yourself, yourselves, zero
</pre>

[[Category: MySQL]]

MySQL

2010-03-06T01:35:59+00:00

← Older revision		Revision as of 01:35, 6 March 2010
Line 51:		Line 51:
	* [[MySQL C API]]		* [[MySQL C API]]
	* [[MySQL Date and Time Functions]]		* [[MySQL Date and Time Functions]]
		+	* [[MySQL Stop Words]]

	== External Links ==		== External Links ==

Squid 3 To Block HTTP Headers

2010-03-06T01:27:46+00:00

← Older revision		Revision as of 01:27, 6 March 2010
Line 38:		Line 38:
	request_header_access Connection allow all		request_header_access Connection allow all
	request_header_access Proxy-Connection allow all		request_header_access Proxy-Connection allow all
		+	request_header_access User-Agent allow all
	request_header_access All allow trusted_domains		request_header_access All allow trusted_domains
	request_header_access All deny all		request_header_access All deny all
Line 68:		Line 69:
	reply_header_access Connection allow all		reply_header_access Connection allow all
	reply_header_access Proxy-Connection allow all		reply_header_access Proxy-Connection allow all
		+	reply_header_access User-Agent allow all
	reply_header_access All allow trusted_domains		reply_header_access All allow trusted_domains
	reply_header_access All deny all		reply_header_access All deny all

Squid

2010-03-05T23:37:33+00:00

← Older revision		Revision as of 23:37, 5 March 2010
Line 17:		Line 17:
	* http://www.deckle.co.za/squid-users-guide/Squid_Configuration_Basics		* http://www.deckle.co.za/squid-users-guide/Squid_Configuration_Basics

-	[[Category:Proxies]]	+	[[Category: Proxies]]
-	[[Category:Caching]]	+	[[Category: Caching]]
		+	[[Category: Squid]]

Category:Squid

2010-03-05T23:37:12+00:00

Created page with 'Squid'

New page

[[Squid]]

Category:Privacy

2010-03-05T22:43:01+00:00

Created page with 'Privacy'

New page

[[Privacy]]

Squid 3 To Block HTTP Headers

2010-03-05T22:42:51+00:00

New page

/etc/squid3/acls/trusted_domains:
<pre>
.google.com
.chase.com
.bankofamerica.com
.savonix.com
.docunext.com
.firstequitycard.com
.pbooks.org
</pre>

Part of squid.conf
<pre>
request_header_access Allow allow all
request_header_access Authorization allow all
request_header_access WWW-Authenticate allow all
request_header_access Proxy-Authorization allow all
request_header_access Proxy-Authenticate allow all
request_header_access Cache-Control allow all
request_header_access Content-Encoding allow all
request_header_access Content-Length allow all
request_header_access Content-Type allow all
request_header_access Date allow all
request_header_access Expires allow all
request_header_access Host allow all
request_header_access If-Modified-Since allow all
request_header_access Last-Modified allow all
request_header_access Location allow all
request_header_access Pragma allow all
request_header_access Accept allow all
request_header_access Accept-Charset allow all
request_header_access Accept-Encoding allow all
request_header_access Accept-Language allow all
request_header_access Content-Language allow all
request_header_access Mime-Version allow all
request_header_access Retry-After allow all
request_header_access Title allow all
request_header_access Connection allow all
request_header_access Proxy-Connection allow all
request_header_access All allow trusted_domains
request_header_access All deny all

reply_header_access Allow allow all
reply_header_access Authorization allow all
reply_header_access WWW-Authenticate allow all
reply_header_access Proxy-Authorization allow all
reply_header_access Proxy-Authenticate allow all
reply_header_access Cache-Control allow all
reply_header_access Content-Encoding allow all
reply_header_access Content-Length allow all
reply_header_access Content-Type allow all
reply_header_access Date allow all
reply_header_access Expires allow all
reply_header_access Host allow all
reply_header_access If-Modified-Since allow all
reply_header_access Last-Modified allow all
reply_header_access Location allow all
reply_header_access Pragma allow all
reply_header_access Accept allow all
reply_header_access Accept-Charset allow all
reply_header_access Accept-Encoding allow all
reply_header_access Accept-Language allow all
reply_header_access Content-Language allow all
reply_header_access Mime-Version allow all
reply_header_access Retry-After allow all
reply_header_access Title allow all
reply_header_access Connection allow all
reply_header_access Proxy-Connection allow all
reply_header_access All allow trusted_domains
reply_header_access All deny all
</pre>

[[Category: Squid]]
[[Category: Caching]]
[[Category: Proxies]]
[[Category: Privacy]]

Squid

2010-03-05T21:35:04+00:00

← Older revision		Revision as of 21:35, 5 March 2010
Line 8:		Line 8:
	</pre>		</pre>

		+	* [[Squid 3 To Block HTTP Headers]]

	== Related Pages ==		== Related Pages ==

docunext pushed to master at docunext/chimailmadmin

2010-03-04T20:54:25+00:00

docunext committed 029716f

added link to docunext dig

docunext pushed to master at docunext/regdel

2010-03-04T20:51:42+00:00

docunext committed ea67881

no xmlns

docunext pushed to master at docunext/chimailmadmin

2010-03-04T20:16:14+00:00

docunext committed e14a240

no xhtml namespace on micro view xsl

Example MongoDB Application

2010-03-04T20:05:16+00:00

In addition to Docunext, I also work on a website called Informed Banking.

Like Docunext, it also has a blog and a wiki, but it also has a bank directory which I just happened to rebuild using Ruby 1.9.1, Sinatra, XSL, Builder, Erubis, Markdown, and MongoDB. The old bank directory used Nexista and MySQL and that worked well for approximately five years. However, it would break from time to time and was difficult to maintain, alter, or update.

The new one was a breeze to create. It took about a week to rebuild and only a couple hours each day. I'm deploying the code with Vlad, which makes a world of difference.

The really cool thing about using MongoDB is how easy it is to filter the data based upon different indices, like state, zip code, or even city and state.

Without further ado, here's a link so you can peruse it and try out the cool new features:

The New and Improved Informed Banking Bank Directory

Enjoy!

docunext pushed to master at docunext/chimailmadmin

2010-03-04T18:21:41+00:00

docunext committed 72b8b93

domain groups stem
docunext committed bacc851

new view

Importing a Large Dataset in MongoDB

2010-03-04T17:48:59+00:00

Last night I imported a large, but not outrageously large, dataset into a MongoDB database. The complication might have been the structure of each document. They we're too complicated, just a hash with two keys, one having a string as a value, and the other having an array.

At first I was using Ruby1.9.1, but it was taking too long so I switched to using the command line interface, "mongoimport".

The cool thing about this is that it can import JSON directly. I converted my data set to JSON format, saved it to a single file with about 23,000 JSON objects, and then ran it:

mongoimport --host 192.168.8.103 --db doculabsappone -c tags  tmp/tags.json

It was way faster than using Ruby1.9.1!

A Really Delicious Ruby Error

2010-03-04T05:21:41+00:00

Created page with '<pre> config.ru:1:in `new' config.ru:1:in `block in <top (required)>' /var/lib/gems/1.9.1/gems/unicorn-0.96.1/bin/unicorn:130:in `eval' /var/lib/gems/1.9.1/gems/unicorn-0.96.1/bi...'

New page

<pre>
config.ru:1:in `new'
config.ru:1:in `block in <top (required)>'
/var/lib/gems/1.9.1/gems/unicorn-0.96.1/bin/unicorn:130:in `eval'
/var/lib/gems/1.9.1/gems/unicorn-0.96.1/bin/unicorn:130:in `block in <top (required)>'
/var/lib/gems/1.9.1/gems/unicorn-0.96.1/lib/unicorn.rb:759:in `call'
/var/lib/gems/1.9.1/gems/unicorn-0.96.1/lib/unicorn.rb:759:in `build_app!'
/var/lib/gems/1.9.1/gems/unicorn-0.96.1/lib/unicorn.rb:215:in `start'
/var/lib/gems/1.9.1/gems/unicorn-0.96.1/lib/unicorn.rb:28:in `run'
/var/lib/gems/1.9.1/gems/unicorn-0.96.1/bin/unicorn:165:in `<top (required)>'
/var/lib/gems/1.9.1/bin/unicorn:19:in `load'
/var/lib/gems/1.9.1/bin/unicorn:19:in `<main>'

-- C level backtrace information -------------------------------------------
0xb80288d4 /usr/lib/libruby-1.9.1.so.1.9(rb_vm_bugreport+0x64) [0xb80288d4]
0xb7f3e169 /usr/lib/libruby-1.9.1.so.1.9 [0xb7f3e169]
0xb7f3e1fa /usr/lib/libruby-1.9.1.so.1.9(rb_bug+0x3a) [0xb7f3e1fa]
0xb7fcb66d /usr/lib/libruby-1.9.1.so.1.9 [0xb7fcb66d]
0xb809740c [0xb809740c]
0xb7f8b6b3 /usr/lib/libruby-1.9.1.so.1.9(ruby_yyparse+0x6393) [0xb7f8b6b3]
0xb7f952ac /usr/lib/libruby-1.9.1.so.1.9 [0xb7f952ac]
0xb802b9db /usr/lib/libruby-1.9.1.so.1.9(ruby_suppress_tracing+0x10b) [0xb802b9db]
0xb7f802af /usr/lib/libruby-1.9.1.so.1.9(rb_parser_compile_file+0x8f) [0xb7f802af]
0xb7fc9aba /usr/lib/libruby-1.9.1.so.1.9 [0xb7fc9aba]
0xb7f3f6d9 /usr/lib/libruby-1.9.1.so.1.9(rb_ensure+0x99) [0xb7f3f6d9]
0xb7fc7d1c /usr/lib/libruby-1.9.1.so.1.9(rb_load_file+0x6c) [0xb7fc7d1c]
0xb7f4224e /usr/lib/libruby-1.9.1.so.1.9(rb_load+0x12e) [0xb7f4224e]
0xb7f42f93 /usr/lib/libruby-1.9.1.so.1.9(rb_require_safe+0x543) [0xb7f42f93]
0xb7f432a6 /usr/lib/libruby-1.9.1.so.1.9(rb_f_require+0x26) [0xb7f432a6]
0xb8013f9b /usr/lib/libruby-1.9.1.so.1.9 [0xb8013f9b]
0xb80142a8 /usr/lib/libruby-1.9.1.so.1.9 [0xb80142a8]
0xb80240a3 /usr/lib/libruby-1.9.1.so.1.9 [0xb80240a3]
0xb80197d8 /usr/lib/libruby-1.9.1.so.1.9 [0xb80197d8]
0xb801dd71 /usr/lib/libruby-1.9.1.so.1.9 [0xb801dd71]
0xb801fc25 /usr/lib/libruby-1.9.1.so.1.9 [0xb801fc25]
0xb8020511 /usr/lib/libruby-1.9.1.so.1.9 [0xb8020511]
0xb8013fc0 /usr/lib/libruby-1.9.1.so.1.9 [0xb8013fc0]
0xb80142a8 /usr/lib/libruby-1.9.1.so.1.9 [0xb80142a8]
0xb80240a3 /usr/lib/libruby-1.9.1.so.1.9 [0xb80240a3]
0xb80197d8 /usr/lib/libruby-1.9.1.so.1.9 [0xb80197d8]
0xb801dd71 /usr/lib/libruby-1.9.1.so.1.9 [0xb801dd71]
0xb8020f3c /usr/lib/libruby-1.9.1.so.1.9 [0xb8020f3c]
0xb8023dfe /usr/lib/libruby-1.9.1.so.1.9(rb_funcall2+0x6e) [0xb8023dfe]
0xb7f3f630 /usr/lib/libruby-1.9.1.so.1.9(rb_obj_call_init+0x50) [0xb7f3f630]
0xb7f7511a /usr/lib/libruby-1.9.1.so.1.9(rb_class_new_instance+0x3a) [0xb7f7511a]
0xb8013fc0 /usr/lib/libruby-1.9.1.so.1.9 [0xb8013fc0]
0xb80142a8 /usr/lib/libruby-1.9.1.so.1.9 [0xb80142a8]
0xb80240a3 /usr/lib/libruby-1.9.1.so.1.9 [0xb80240a3]
0xb80197d8 /usr/lib/libruby-1.9.1.so.1.9 [0xb80197d8]
0xb801dd71 /usr/lib/libruby-1.9.1.so.1.9 [0xb801dd71]
0xb801ec78 /usr/lib/libruby-1.9.1.so.1.9 [0xb801ec78]
0xb801f2e6 /usr/lib/libruby-1.9.1.so.1.9(rb_f_eval+0xf6) [0xb801f2e6]
0xb8013fc0 /usr/lib/libruby-1.9.1.so.1.9 [0xb8013fc0]
0xb80142a8 /usr/lib/libruby-1.9.1.so.1.9 [0xb80142a8]
0xb80240a3 /usr/lib/libruby-1.9.1.so.1.9 [0xb80240a3]
0xb80197d8 /usr/lib/libruby-1.9.1.so.1.9 [0xb80197d8]
0xb801dd71 /usr/lib/libruby-1.9.1.so.1.9 [0xb801dd71]
0xb80209c4 /usr/lib/libruby-1.9.1.so.1.9(rb_vm_invoke_proc+0x294) [0xb80209c4]
0xb7f44bd3 /usr/lib/libruby-1.9.1.so.1.9 [0xb7f44bd3]
0xb8013fc0 /usr/lib/libruby-1.9.1.so.1.9 [0xb8013fc0]
0xb80142a8 /usr/lib/libruby-1.9.1.so.1.9 [0xb80142a8]
0xb80240a3 /usr/lib/libruby-1.9.1.so.1.9 [0xb80240a3]
0xb80197d8 /usr/lib/libruby-1.9.1.so.1.9 [0xb80197d8]
0xb801dd71 /usr/lib/libruby-1.9.1.so.1.9 [0xb801dd71]
0xb801e0ed /usr/lib/libruby-1.9.1.so.1.9(rb_iseq_eval+0x12d) [0xb801e0ed]
0xb7f422a7 /usr/lib/libruby-1.9.1.so.1.9(rb_load+0x187) [0xb7f422a7]
0xb7f423f7 /usr/lib/libruby-1.9.1.so.1.9 [0xb7f423f7]
0xb8013fc0 /usr/lib/libruby-1.9.1.so.1.9 [0xb8013fc0]
0xb80142a8 /usr/lib/libruby-1.9.1.so.1.9 [0xb80142a8]
0xb80240a3 /usr/lib/libruby-1.9.1.so.1.9 [0xb80240a3]
0xb80197d8 /usr/lib/libruby-1.9.1.so.1.9 [0xb80197d8]
0xb801dd71 /usr/lib/libruby-1.9.1.so.1.9 [0xb801dd71]
0xb801df9b /usr/lib/libruby-1.9.1.so.1.9(rb_iseq_eval_main+0xab) [0xb801df9b]
0xb7f40297 /usr/lib/libruby-1.9.1.so.1.9(ruby_exec_node+0xb7) [0xb7f40297]
0xb7f41896 /usr/lib/libruby-1.9.1.so.1.9(ruby_run_node+0x56) [0xb7f41896]
0x80487b4 /usr/bin/ruby1.9.1(main+0x74) [0x80487b4]
0xb7d2d775 /lib/i686/cmov/libc.so.6(__libc_start_main+0xe5) [0xb7d2d775]
0x80486a1 /usr/bin/ruby1.9.1 [0x80486a1]
</pre>

Ruby Errors

2010-03-04T05:21:35+00:00

Created page with '* A Really Delicious Ruby Error Category: Ruby'

New page

* [[A Really Delicious Ruby Error]]

[[Category: Ruby]]

Ruby

2010-03-04T05:21:14+00:00

Interesting Ruby Stuff:

← Older revision		Revision as of 05:21, 4 March 2010
Line 93:		Line 93:
	* [[Ruby Rip]]		* [[Ruby Rip]]
	* [[Jscmd]]		* [[Jscmd]]
		+	* [[Ruby Errors]]

	== External Links ==		== External Links ==

Roundcube

2010-03-04T05:03:42+00:00

News:

← Older revision		Revision as of 05:03, 4 March 2010
Line 3:		Line 3:

	==News==		==News==
-	* [http://www.docunext.com/blog/2007/12~~/09~~/roundcube-on-debian/ Roundcube is now available on debian!!]	+	* [http://www.docunext.com/blog/2007/12/roundcube-on-debian.html Roundcube is now available on debian!!]
-		+

	==My Experience==		==My Experience==

Xen

2010-03-04T04:17:20+00:00

← Older revision		Revision as of 04:17, 4 March 2010
Line 9:		Line 9:

	I recently installed [[Ubuntu]] on a Mac Mini and the experience was so wonderful, I might try using Ubuntu and another Mac Mini I have for my first Xen machine. Its pretty cool that Mac mini's have a [[CPU]] that supports Xen. Its important to note that Ubuntu does not include a dom0 capable kernel in their repositories, but apparently a dom0 capable kernel from [[debian]] can be used without any issues.		I recently installed [[Ubuntu]] on a Mac Mini and the experience was so wonderful, I might try using Ubuntu and another Mac Mini I have for my first Xen machine. Its pretty cool that Mac mini's have a [[CPU]] that supports Xen. Its important to note that Ubuntu does not include a dom0 capable kernel in their repositories, but apparently a dom0 capable kernel from [[debian]] can be used without any issues.
		+
		+
		+	== Old Xen Page ==
		+	I kept noticing that [[debian]] has a stock kernel for xen, and then I realized a [[GNU/Linux]] xen host could run a [[FreeBSD]] guest, and now I'm really interested.
		+
		+	I'm finally trying out Xen on debian.
		+
		+
		+

	== External Links ==		== External Links ==

XEN

2010-03-04T04:17:01+00:00

Redirected page to Xen

← Older revision		Revision as of 04:17, 4 March 2010
Line 1:		Line 1:
-	~~I kept noticing that~~ [[~~debian~~]] ~~has a stock kernel for xen, and then I realized a [[GNU/Linux]] xen host could run a [[FreeBSD]] guest, and now I'm really interested.~~	+	#REDIRECT [[Xen]]
-		+
-	~~I'm finally trying out Xen on debian.~~	+
-		+
-		+
-	~~== External Links ==~~	+
-	* http://wiki.debian.org/DebianInstaller/Xen	+

Using jQuery 1.4+ to Load CSS

2010-03-04T02:29:16+00:00

Foreward

I think these are features new to jQuery 1.4, but I'm certain they are new to me.

In the past, I've often wondered the best way to create new HTML with jQuery. I have a way I like to do it with raw javascript, but I only employ that when automatically generating the code.

With jQuery, unless I'm loading in data from an external resource, I usually only need to add a little HTML; a single node for instance.

Let's look at the old, recent and new ways I've done and am doing this.

Old Way

if($('.thickbox').length > 0) {
  $.getScript('/s/js/jquery/plugins/thickbox-compressed.js', function() {
    $('head').append('<link rel="stylesheet" type="text/css" href="/s/css/thickbox.min.css" />');
  });
}

One problem here is that I don't think its not compatible with both HTML and XHTML. Its also pretty ugly, too.

Recent Way

I read up on using jQuery to load CSS and found something like following. Better, huh?

if($('.thickbox').length > 0) {
  $.getScript('/s/js/jquery/plugins/thickbox-compressed.js', function() {
    $('head').append('<link>');
    css = $('head').children(':last');
    css.attr({
      rel:  'stylesheet',
      type: 'text/css',
      href: '/s/css/thickbox.min.css'
    });
  });
}

I like this because it uses more javascript, but it requires two updates to the DOM.

New Way

This is what I think is new with jQuery 1.4+. Its the ability to create a new node internally, modify it, and then add it to the DOM. This might have been possible with jQuery 1.3, but I wasn't aware of it. Regardless, its cool!

if($('.thickbox').length > 0) {
  $.getScript('/s/js/jquery/plugins/thickbox-compressed.js', function() {
    $('<link>', {
      'rel':  'stylesheet',
      'type': 'text/css',
      'href': '/s/css/thickbox.min.css'
    }).appendTo('head');
  });
}

I'm not done with this yet. I'm planning to make it into a function with the CSS href as the argument.

Redis-rb

2010-03-04T02:02:19+00:00

← Older revision		Revision as of 02:02, 4 March 2010
Line 26:		Line 26:
	* [[Redis]]		* [[Redis]]

		+	== External Links ==
		+	* http://blog.grayproductions.net/articles/lists_and_sets_in_redis

	[[Category: NoSQL]]		[[Category: NoSQL]]
	[[Category: Redis]]		[[Category: Redis]]
	[[Category: Ruby]]		[[Category: Ruby]]

Lists

2010-03-04T02:01:59+00:00

← Older revision		Revision as of 02:01, 4 March 2010
Line 4:		Line 4:


-	== External ~~Lists~~ ==	+	== External Links ==
-	* http://www.census.gov/genealogy/names/~~dist~~.~~all.last~~	+	* http://www.census.gov/genealogy/names/names_files.html

	[[Category: Lists]]		[[Category: Lists]]

Redis-rb incr-decr.rb

2010-03-04T01:47:26+00:00

Created page with '<pre class="sh_ruby"> require 'rubygems' require 'redis' r = Redis.new puts p 'incr' r.delete 'counter' p r.incr('counter') p r.incr('counter') p r.incr('counter') puts p 'de...'

New page

<pre class="sh_ruby">
require 'rubygems'
require 'redis'

r = Redis.new

puts
p 'incr'
r.delete 'counter'

p r.incr('counter')
p r.incr('counter')
p r.incr('counter')

puts
p 'decr'
p r.decr('counter')
p r.decr('counter')
p r.decr('counter')
</pre>

Redis-rb

2010-03-04T01:47:15+00:00

Examples from the Source Code:

← Older revision		Revision as of 01:47, 4 March 2010
Line 20:		Line 20:
	* [[Redis-rb list.rb]]		* [[Redis-rb list.rb]]
	* [[Redis-rb sets.rb]]		* [[Redis-rb sets.rb]]
		+	* [[Redis-rb incr-decr.rb]]

	== Related Pages ==		== Related Pages ==

Redis-rb list.rb

2010-03-04T00:39:29+00:00

← Older revision		Revision as of 00:39, 4 March 2010
Line 27:		Line 27:
	p r.list_range('logs', 0, -1)		p r.list_range('logs', 0, -1)
	</pre>		</pre>
		+
		+
		+	[[Category: Redis]]

Redis-rb sets.rb

2010-03-04T00:39:14+00:00

New page

<pre class="sh_ruby">
require 'rubygems'
require 'redis'

r = Redis.new

r.delete 'foo-tags'
r.delete 'bar-tags'

puts
p "create a set of tags on foo-tags"

r.set_add 'foo-tags', 'one'
r.set_add 'foo-tags', 'two'
r.set_add 'foo-tags', 'three'

puts
p "create a set of tags on bar-tags"

r.set_add 'bar-tags', 'three'
r.set_add 'bar-tags', 'four'
r.set_add 'bar-tags', 'five'

puts
p 'foo-tags'

p r.set_members('foo-tags')

puts
p 'bar-tags'

p r.set_members('bar-tags')

puts
p 'intersection of foo-tags and bar-tags'

p r.set_intersect('foo-tags', 'bar-tags')
</pre>

[[Category: Redis]]

Redis-rb

2010-03-04T00:34:29+00:00

Examples from the Source Code:

← Older revision		Revision as of 00:34, 4 March 2010
Line 19:		Line 19:

	* [[Redis-rb list.rb]]		* [[Redis-rb list.rb]]
		+	* [[Redis-rb sets.rb]]

	== Related Pages ==		== Related Pages ==

Redis-rb list.rb

2010-03-04T00:20:22+00:00

Created page with '<pre class="sh_ruby"> require 'rubygems' require 'redis' r = Redis.new r.delete 'logs' puts p "pushing log messages into a LIST" r.push_tail 'logs', 'some log message' r.push...'

New page

<pre class="sh_ruby">
require 'rubygems'
require 'redis'

r = Redis.new

r.delete 'logs'

puts

p "pushing log messages into a LIST"
r.push_tail 'logs', 'some log message'
r.push_tail 'logs', 'another log message'
r.push_tail 'logs', 'yet another log message'
r.push_tail 'logs', 'also another log message'

puts
p 'contents of logs LIST'

p r.list_range('logs', 0, -1)

puts
p 'Trim logs LIST to last 2 elements(easy circular buffer)'

r.list_trim('logs', -2, -1)

p r.list_range('logs', 0, -1)
</pre>

Redis-rb

2010-03-04T00:20:11+00:00

Documentation:

← Older revision		Revision as of 00:20, 4 March 2010
Line 16:		Line 16:


		+	== Examples from the Source Code ==
		+
		+	* [[Redis-rb list.rb]]

	== Related Pages ==		== Related Pages ==

docunext pushed to master at docunext/Rack-XSLView

2010-03-03T22:02:14+00:00

docunext committed e7c6470

updated docs, added reload capability
docunext committed bf827c4

updated docs, added reload capability

docunext pushed to master at docunext/Rack-XSLView

2010-03-03T21:38:26+00:00

docunext committed c4d09e6

option to reload xsl every time, bumping version

MongoDB Limit Results with Ruby

2010-03-03T19:32:58+00:00

This is similar to the LIMIT function of MySQL (or SQL in general):

coll.find({},{:skip => 0, :limit => 10 })

Reviews of Redis and MongoDB

2010-03-02T21:09:37+00:00

Now that I've actually used both Redis and MongoDB a little, I have to say I really like them both a lot. In tandem with using these two, I also worked with MemcacheDB and Tokyo Tyrant. I like those too, but I want to write about Redis and MongoDB because they have substantially different features than what I've been familiar with in the past.

Redis

For me, the absolute most awesome feature about Redis is the native list data type. I'm already thinking about all the lists I'm going to create:

Server logs
Software packages
States
Colors
Hardware manufacturers

Yes, I'm really looking forward to having that information readily available!

The fact that lists can be truncated makes circular logging super easy too!

MongoDB

Oooh... I'm not sure where to start on this. I've setup a little blog engine on Docunext Labs using MongoDB and I'm impressed! MongoDB is:

Fast
Flexible
Easy to use

It has a nice "vibe" to it. :-)

List of All 50 United States

2010-03-02T20:30:59+00:00

← Older revision		Revision as of 20:30, 2 March 2010
Line 32:		Line 32:
	New Mexico		New Mexico
	New York		New York
-	North	+	North Carolina
-	Carolina	+	North Dakota
-	North	+
-	Dakota	+
	Ohio		Ohio
	Oklahoma		Oklahoma
	Oregon		Oregon
	Pennsylvania		Pennsylvania
-	Rhode	+	Rhode Island
-	Island	+	South Carolina
-	South	+	South Dakota
-	Carolina	+
-	South	+
-	Dakota	+
	Tennessee		Tennessee
	Texas		Texas
Line 52:		Line 47:
	Virginia		Virginia
	Washington		Washington
-	West	+	West Virginia
-	Virginia	+
	Wisconsin		Wisconsin
	Wyoming		Wyoming

Lists

2010-03-02T20:30:15+00:00

New page

Docunext has some lists which may be useful.

* [[List of All 50 United States]]

[[Category: Lists]]

Category:Lists

2010-03-02T20:01:55+00:00

Created page with 'The lists category featured simple lists which may be helpful.'

New page

The lists category featured simple lists which may be helpful.

List of All 50 United States

2010-03-02T20:00:54+00:00

← Older revision		Revision as of 20:00, 2 March 2010
Line 57:		Line 57:
	Wyoming		Wyoming
	</pre>		</pre>
		+
		+
		+	[[Category: Lists]]

docunext pushed to master at docunext/notasinatraapp

2010-03-02T19:37:25+00:00

docunext committed b94fca9

changed mongo db ref from db to mgdb

Incomplete dpkg index

2010-03-02T18:19:43+00:00

← Older revision		Revision as of 18:19, 2 March 2010
Line 679:		Line 679:
	]		]
	</pre>		</pre>
		+
		+	== Related Pages ==
		+	* [[DPKG]]
		+	* [[Debian]]
		+
		+	[[Category: Dpkg]]

Rpm

2010-03-02T18:13:24+00:00

Redirected page to RPM

New page

#REDIRECT [[RPM]]

Ruby Xapian Bindings

2010-03-02T18:09:53+00:00

← Older revision		Revision as of 18:09, 2 March 2010
Line 4:		Line 4:

	* [[Xapian Ruby simpleindex.rb]]		* [[Xapian Ruby simpleindex.rb]]
		+
		+
		+
		+	== External Links ==
		+	* http://www.ruby.code-experiments.com/blog/2010/03/ruby-xapian-gem.html
		+
		+	[[Category: Ruby]]
		+	[[Category: Xapian]]

Xapian Ruby simpleindex.rb

2010-03-02T18:09:01+00:00

Created page with '<pre class="sh_ruby"> #!/usr/bin/env ruby # # Index each paragraph of a text file as a Xapian document. # # Originally by Paul Legato (plegato@nks.net), 4/22/06 # Based on Python...'

New page

<pre class="sh_ruby">
#!/usr/bin/env ruby
#
# Index each paragraph of a text file as a Xapian document.
#
# Originally by Paul Legato (plegato@nks.net), 4/22/06
# Based on Python's simplesearch.py
# Copyright (C) 2006 Networked Knowledge Systems, Inc.
# Copyright (C) 2007 Olly Betts
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2 of the
# License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301
# USA

require 'xapian'

if ARGV.size != 1
$stderr.puts "Usage: #{$0} PATH_TO_DATABASE"
exit 99
end

# Open the database for update, creating a new database if necessary.
database = Xapian::WritableDatabase.new(ARGV[0], Xapian::DB_CREATE_OR_OPEN)

indexer = Xapian::TermGenerator.new()
stemmer = Xapian::Stem.new("english")
indexer.stemmer = stemmer

para = ''
while line = $stdin.gets()
line.strip!()
if line.empty?
if not para.empty?
# We've reached the end of a paragraph, so index it.
doc = Xapian::Document.new()
doc.data = para

indexer.document = doc
indexer.index_text(para)

# Add the document to the database
database.add_document(doc)
para = ''
end # if not para.empty?
else # line not empty
para += ' ' if para != ''
para += line
end # if line empty
end

</pre>

Ruby Xapian Bindings

2010-03-02T18:08:50+00:00

Created page with 'You can interface with Xapian from Ruby! == Examples from Xapian.org == * Xapian Ruby simpleindex.rb'

New page

You can interface with [[Xapian]] from [[Ruby]]!

== Examples from [[Xapian]].org ==

* [[Xapian Ruby simpleindex.rb]]

Xapian

2010-03-02T18:07:50+00:00

← Older revision		Revision as of 18:07, 2 March 2010
Line 4:		Line 4:
	== Related Pages ==		== Related Pages ==
	* [[apt-xapian-index]]		* [[apt-xapian-index]]
		+	* [[Ruby Xapian Bindings]]

	== External Links ==		== External Links ==
Line 9:		Line 10:


-	[[Category:Search]]	+	[[Category: Search]]
		+	[[Category: Xapian]]

Category:Xapian

2010-03-02T18:07:29+00:00

Created page with 'Xapian'

New page

[[Xapian]]

Apt-xapian-index

2010-03-02T18:07:22+00:00

New page

<pre>
===============
Database layout
===============

This Xapian database indexes Debian package information. To query the
database, open it as ``/var/lib/apt-xapian-index/index``.

Data are indexed either as terms or as values. Words found in package
descriptions are indexed lowercase, and all other kinds of terms have an
uppercase prefix as documented below.

Numbers are indexed as Xapian numeric values. A list of the meaning of the
numeric values is found in ``/var/lib/apt-xapian-index/values``.

The data sources used for indexing are:

* Apt tags: Debtags tag information from the Packages file
* Package descriptions: terms extracted from the package descriptions using Xapian's TermGenerator
* Package sections: Debian package sections
* Sizes: package sizes indexed as values

This Xapian index follows the conventions for term prefixes described in
``/usr/share/doc/xapian-omega/termprefixes.txt.gz``.

Extra Debian data sources can define more extended prefixes (starting with
``X``): their meaning is documented below together with the rest of the data
source documentation.

At the very least, at least the package name (with the ``XP`` prefix) will
be present in every document in the database. This allows to quickly
lookup a Xapian document by package name.

The user data associated to a Xapian document is the package name.

-------------------
Active data sources
-------------------

Apt tags
========

The Apt tags data source indexes Debtags tags as found in the
Packages file as terms with the ``XT`` prefix; for example:
'XTrole::program'.

Using the ``XT`` terms, queries can be enhanced with semantic
information. Xapian's support for complex expressions in queries
can be used to great effect: for example::

XTrole::program AND XTuse::gameplaying AND (XTinterface::x11 OR XTinterface::3d)

``XT`` terms can also be used to improve the quality of search
results. For example, the ``gimp`` package would not usually show
up when searching the terms ``image editor``. This can be solved
using the following technique:

1. Perform a normal query
2. Put the first 5 or so results in an Rset
3. Call Enquire::get_eset using the Rset and an expand filter that
only accepts ``XT`` terms. This gives you the tags that are
most relevant to the query.
4. Add the resulting terms to the initial query, and search again.

The Apt tags data source will not work when Debtags is installed,
as Debtags is able to provide a better set of tags.

Package descriptions
====================

The Descriptions data source simply uses Xapian's TermGenerator to
tokenise and index the package descriptions.

Currently this creates normal terms as well as stemmed terms
prefixed with ``Z``.

Package sections
================

The section is indexed literally, with the prefix XS.

Sizes
=====

The Sizes data source indexes the package size and the installed
size as the ``packagesize`` and ``installedsize`` Xapian values.
</pre>

[[Category: Xapian]]
[[Category: Debian]]
[[Category: Apt]]

Xapian

2010-03-02T17:56:04+00:00

← Older revision		Revision as of 17:56, 2 March 2010
Line 1:		Line 1:
	Xapian looks really cool, its an open source [[C++]] search engine, with bindings for a lot of languages (including [[PHP]]!). It seems like its simpler to install and setup than [[leyna]] which looks cool too but more than I would need.		Xapian looks really cool, its an open source [[C++]] search engine, with bindings for a lot of languages (including [[PHP]]!). It seems like its simpler to install and setup than [[leyna]] which looks cool too but more than I would need.

		+
		+	== Related Pages ==
		+	* [[apt-xapian-index]]

	== External Links ==		== External Links ==

Ruby Xapian Gem

2010-03-02T17:47:06+00:00

I'm installing the Ruby Xapian gem, an interface to the Xapian indexing system:

sudo gem1.9.1 install xapian
Building native extensions.  This could take a while...

Nope, that didn't work:

*** Building bindings for languages: ruby

configure: WARNING: Xapian library is version 1.0.18 but the bindings are version 1.0.15 - we strongly recommend using matching versions.

This helped:

Setup Xapian and Ruby binding on Ubuntu server

I'm using Xapian 1.0.18, Ruby 1.9.1, and I excluded other language bindings:

dev-48-gl:/var/www/dev/xapian-bindings/xapian-bindings-1.0.18$ ./configure RUBY=/usr/bin/ruby1.9.1 --without-php --without-python --without-tcl --without-csharp --without-java
make
sudo make install

irb(main):003:0> require 'xapian'
=> true
irb(main):004:0> quit

Nice!

What will I be indexing? Just apt data to start, thanks to "apt-xapian-index".

Hah, I just noticed that the page by Taylor Luk also mentions the use of Xapian with CouchDB. Cool! I've chosen MongoDB over CouchDB, and it looks like there is interest in bridging MongoDB with Xapian, too. Cool!

External Links

http://xapian.org/docs/bindings/ruby/

Redis

2010-03-02T00:51:55+00:00

← Older revision		Revision as of 00:51, 2 March 2010
(One intermediate revision not shown)
Line 1:		Line 1:
	Redis is a persistent key-value store system written in ANSI-C.		Redis is a persistent key-value store system written in ANSI-C.

-	Looks kind of like [[MemcacheDB]], with slightly more storage capabilities. Although only slightly, the little more might make all the difference. Actually, yes! The native '''~~sets~~''' capability of Redis is pretty amazing. How else would it be possible to have access to an array of thousands of items? With [[MemcacheDB]] or [[Tokyo Tyrant]] the array would have to be serialized as a string. Wait - I was just reading up on it a little bit, and the list is stored as a string, but it can be appended to without having to read the string first, i.e. the server can do the appending. Still cool.	+	Looks kind of like [[MemcacheDB]], with slightly more storage capabilities. Although only slightly, the little more might make all the difference. Actually, yes!
		+
		+	== Data Types ==
		+	=== Lists ===
		+	The native '''list''' capability of Redis is pretty amazing. How else would it be possible to have access to an array of thousands of items? With [[MemcacheDB]] or [[Tokyo Tyrant]] the array would have to be serialized as a string. Wait - I was just reading up on it a little bit, and the list is stored as a string, but it can be appended to without having to read the string first, i.e. the server can do the appending. Still cool. Plus, the docs add this:
		+
		+	<blockquote>
		+	The max length of a list is 2^32-1 elements (4294967295, more than 4 billion of elements per list).
		+	</blockquote>
		+
		+	Sweet!

	== Redis Configuration ==		== Redis Configuration ==

Redis-rb

2010-03-02T00:45:52+00:00

← Older revision		Revision as of 00:45, 2 March 2010
(2 intermediate revisions not shown)
Line 1:		Line 1:
-	[[Redis]] [[Ruby]] client library...	+	[[Redis]] [[Ruby]] client library.

	== Documentation ==		== Documentation ==
	Redis-rb documentation seems to be a bit sparse, so I'm putting some of my notes here:		Redis-rb documentation seems to be a bit sparse, so I'm putting some of my notes here:

		+	=== Connection ===
	<pre class="sh_ruby">		<pre class="sh_ruby">
-	rsc = Redis.new(:host => '192.168.8.~~103~~', :db => '~~doculabsappone~~')	+	rsc = Redis.new(:host => '192.168.0.1', :db => 'mydb')
	</pre>		</pre>
		+
		+	=== Sets ===
		+	This produces an array:
		+	<pre class="sh_ruby">
		+	Dlabzapp1.runtime['rdsc'].set_members('dpkgs')
		+	</pre>
		+
		+

	== Related Pages ==		== Related Pages ==

Converting MemcacheDB Posts to MongoDB

2010-03-01T23:16:31+00:00

This probably won't make much sense out of context, but I wanted to make note of this anyway.

I started work on a little blog engine to learn more about NoSQL, more specifically, publishing with MemcacheDB. I later decided I wanted to use MongoDB. To migrate the old posts to the new storage backend, I wrote this little convert function within my Sinatra App to do so:

    # Temp func to convert memc posts to new mongodb posts
    get '/news/convert' do
      coll = Dlabzapp1.runtime['mgdb'].collection('entries')
      @rawst = Dlabzapp1.memcdb.get('index');
      index = JSON.parse(@rawst)
      index.each_pair do |key,value|
        cnt = Dlabzapp1.memcdb.get(value).to_s
        coll.insert({
          :rurl => value,
          :entry_title => cnt.gsub(/\n\n.+/,''),
          :entry_text => cnt.gsub(/^[^\n]+\n\n/,''),
          :created_on => Time.now.to_i
        })
      end
    end

Then once I confirmed they had been migrated, I deleted them:

    # Temp func to delete memc posts to new mongodb posts
    get '/news/delete' do
      @rawst = Dlabzapp1.memcdb.get('index');
      index = JSON.parse(@rawst)
      result = ""
      index.each_pair do |key,value|
        if Dlabzapp1.memcdb.get(value)
          Dlabzapp1.memcdb.delete(value)
          result  "#{value} has been deleted\n"
        end
      end
      result
    end

I ran this twice. First it reported that the entries had been deleted. Then it reported nothing, as the entries were not found. Then I ran this to delete the index:

    # Temp func to delete memc post index to new mongodb posts
    get '/news/delindex' do
      @rawst = Dlabzapp1.memcdb.delete('index');
      "ok"
    end

Debian and NoSQL Storage Systems

2010-03-01T19:18:44+00:00

After reviewing a bunch of "NoSQL" style document-based / key-value storage systems, I decided to move forward with four of them and install them on production and demonstration machines.

On the production machines, I stuck with Debian's "testing" repository, currently dubbed "Squeeze". As such, I installed MemcacheDB and Redis.

On a demonstration machine, I temporarily added the unstable repository, "Sid", as well as the MongoDB repository for the stable Debian distribution, "Lenny" (which required I add the official Lenny repository as well). With these extra repositories, I was able to install tokyotyrant and mongodb. :-)

Web Servers

2010-03-01T18:09:39+00:00

= Q1 2010:

← Older revision		Revision as of 18:09, 1 March 2010
(One intermediate revision not shown)
Line 47:		Line 47:
	* Port 6081 - Caching proxies, [[ESI]], and initial request routing: [[Varnish]]!		* Port 6081 - Caching proxies, [[ESI]], and initial request routing: [[Varnish]]!
	* Port 80 - Global static files, simple virtual hosts, connection management, compression, and delivery to end-user: [[NGINX]].		* Port 80 - Global static files, simple virtual hosts, connection management, compression, and delivery to end-user: [[NGINX]].
		+
		+
		+	==== Q1 2010 ====
		+	I'm also using separate servers for different application types:
		+
		+	* Standard applications that are packaged from [[Debian]], like [[Movable Type]] and [[MediaWiki]]
		+	* Custom applications that are not packaged but deployed using [[Vlad]]

	== Popular Open Source Web Servers ==		== Popular Open Source Web Servers ==

Apt

2010-03-01T18:05:30+00:00

← Older revision		Revision as of 18:05, 1 March 2010
Line 28:		Line 28:
	* [[Perl]]		* [[Perl]]

-	[[Category:Packaging]]	+	== External Links ==
-	[[Category:Debian]]	+	* http://wiki.debian.org/AptPreferences
		+
		+	[[Category: Packaging]]
		+	[[Category: Debian]]
		+	[[Category: Apt]]
		+	[[Category: Dpkg]]

Regex to Flatten CSS Files

2010-02-28T18:14:10+00:00

tt { border:1px solid #EEEEEE; font-family:courier,mono; font-size:75%; line-height:80%; padding:1px 2px 0; vertical-align:2px; } I'm using jEdit to flatten some CSS files.

I search for this: (;|{)\n and replace with this: $1 (note the extra space after the $1 substitution.

By "flat", I refer to this:

.widget { position:relative; overflow:hidden; width:100%; }

instead of this:

.widget {
  position:relative;
  overflow:hidden;
  width:100%;
}

docunext pushed to master at docunext/phunkybb

2010-02-28T05:14:27+00:00

docunext committed e1bf1b5

fixed number of views sql

Movable Type

2010-02-28T03:33:07+00:00

Movable Type Errors:

← Older revision		Revision as of 03:33, 28 February 2010
Line 60:		Line 60:

	Hope someone is listening!		Hope someone is listening!
		+
		+
		+	== Movable Type 4.3+ ==
		+	Movable Type version 4.3 was a major speed increase from previous versions. [[FastCGI]] support improved substantially.
		+

Movable Type Custom Asset Markup

2010-02-28T03:31:34+00:00

New page

This is a really awesome plugin! It allows you to customize the HTML which is created when assets are uploaded and included in pages and entries.

[[Category: Movable Type]]
[[Category: MTOS]]

Movable Type

2010-02-28T03:28:19+00:00

External:

← Older revision		Revision as of 03:28, 28 February 2010
Line 89:		Line 89:
	<ul><li><a href="http://www.informedblogging.com/">Informed Blogging</a></li></ul>		<ul><li><a href="http://www.informedblogging.com/">Informed Blogging</a></li></ul>
	</html>		</html>
		+	* http://www.mirrors.docunext.com/websvn/movabletypeopensource
	* http://github.com/docunext/docunext-mtos-templates		* http://github.com/docunext/docunext-mtos-templates

-	[[Category:Blogs]]	+	[[Category: Blogs]]
-	[[Category:Perl]]	+	[[Category: Perl]]
-	[[Category:Movable Type]]	+	[[Category: Movable Type]]