How to stop Courier from sending Cannot display Unicode content on pop3

last week we upgraded our webserver from buster to bullseye. within this, our mailserver courier was updated from version 0.76.3-5 to 1.0.16. we downgraded a year or more ago when we did upgrade from stretch to buster, cause we was running into the same problem after the uprade: mails, that were retrieved via pop3 always was “corrupted” cause courier did send a warning message with the subject cannot display unicode and following content:

This E-mail message was determined to be Unicode-formatted
but your E-mail reader does not support Unicode E-mail.

Please use an E-mail reader that supports POP3 with UTF-8
(see https://tools.ietf.org/html/rfc6856.html).

This can also happen when the sender's E-mail program does not
correctly format the sent message.

The original message is included as a separate attachment
so that it can be downloaded manually

after the try to downgrade again to 0.76 which leaded to dependencie problems cause we use mysql for authentication and i didnt wanted to downgrade all the mysql-stuff, studieing the changelogs of courier, change all the last customers from pop3 to imap, cause i couldnt find another solution, i decided to drop Sam, who is one of the main developer of courier, a mail. all the fixes he did relatet to this unicode-alerts were made only for IMAP. he confirmed in his mail, that there were no improvements or changes for pop3 in the last years, and he will not implement some, cause mailclients should not send binary garbage. after some mail-exchanges, he told me that inside the junk-header, the content-preview contains the binary garbage, that causes this alerts. this content-preview is generated by spamassassin. so i stopped spamassassin, send a test mail, and voilat: the problem disappeared! so if one of you will run into the same problem, here is the solution: temporarly i disabeld the spam-report (commenting add_header = X-New-Spam-Report: $acl_m_spam_report) in our acl.spamassassin.conf, that we use with exim.

i will try to fix the broken character-transmitting inside the report when i have more time. will update this post than!

So, here is the update

Following Solution:

in our main.conf

#UTF8 Domain PERL Stuf
allow_utf8_domains = true
perl_startup = do 'FSC_CENTER_DIR/exim/perl/exim.pl'
perl_at_start = true

in this exim.pl

sub header_encode {
	$headerName = shift;
	$headerValue = shift;
	
	$dir = dirname(__FILE__);
#	$eHeaderName = encode_base64($headerName);
#	$eHeaderValue = encode_base64($headerValue);

		#open ( DEBUG, ">>/tmp/header_encode_debug.txt" );
                #print DEBUG "PERL IN: " . $header . "\n";
                #print DEBUG "PERL EXEC: /usr/bin/php $dir/exim.php \"header_encode\" $eHeaderName $eHeaderValue\n";
                #close ( DEBUG );

	$newheader = `/usr/bin/php $dir/exim.php "header_encode" '$headerName' '$headerValue'`;
	
	return ($newheader);
}

…so cause we cant code PERL we just call a php script to do the final encoding.

<?php
ob_start();
error_reporting(E_ALL);

$returnCode = 1;

$todo = isset($_SERVER['argv'][1]) ? $_SERVER['argv'][1] : null;

exec('echo ' . escapeshellarg('PHP TODO: ' . escapeshellarg($todo)) . ' >> /tmp/header_encode_debug.txt');
exec('echo ' . escapeshellarg('PHP IN: ' . escapeshellarg(json_encode($_SERVER['argv']))) . ' >> /tmp/header_encode_debug.txt');


if($todo == 'puny') {
	include(__DIR__.'/IDNAConvert.class.php');

	$address = $_SERVER['argv'][2];
	//$local = substr($address, 0, strpos($address, '@'));
	$domain = $address; //strtolower(substr($address, strpos($address, '@') + 1));

	$returnCode = 0;

	$convert = new IDNAConvert();
	$punycode = $convert->encode($domain);
	if($punycode == null) {
		$address = '';
		$returnCode = 2;
	}
	else {
		$address = $punycode; //"$local@$punycode";
		$returnCode = 0;

		if(!preg_match('/^[a-z0-9\.-]+$/i', $punycode)) {
			$address = '';
			$returnCode = 3;
		}
	}

	echo $address;

	//exec('echo "PHP IN: \"' . $address . '\"" >> /tmp/punycode_debug.txt');
	//exec('echo "PHP PUNY: \"' . $domain . '\" -> \"' . $punycode . '\"" >> /tmp/punycode_debug.txt');
	//exec('echo "PHP RES: \"' . $address . '\" -> \"' . ob_get_contents() . '\"" >> /tmp/punycode_debug.txt');
}
if($todo == 'header_encode') {
	require_once('phar://'.__DIR__.'/FISBMail.2.phar/index.php');

	$headerName = $_SERVER['argv'][2];

	$headerValue = base64_decode($_SERVER['argv'][3]);

	$ho = BMailHeader::create($headerName, $headerValue);

		#file_put_contents("/tmp/header_encode_debug.txt", "PHP IN:\n$headerName: $headerValue\n", FILE_APPEND);

	$result = $ho->getHeaderLinesData();

		#file_put_contents("/tmp/header_encode_debug.txt", "PHP RESULT:\n$result\n", FILE_APPEND);

	$result = "$headerName: " . trim($result);

		#file_put_contents("/tmp/header_encode_debug.txt", "PHP RESULT:\n$result\n", FILE_APPEND);

	echo base64_encode(str_replace("\r\n", "\n", $result));

	$returnCode = 0;
}

die($returnCode);

and finally use it, under our system.filter we added:

if $acl_m_addspamreportheader is not "" then
headers remove X-Spam-Report
headers add ${base64d:${perl{header_encode}{X-Spam-Report}{${base64:$acl_m_addspamreportheader}}}}
endif

thats all, hope this helps someone 🙂

0 0 votes
Article Rating
Abonnieren
Benachrichtige mich bei
2 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Gerald Kainz
10 Monate zuvor

Hi, thanks so much for your investigation dealing with the courier pop problem !
I am having this here on my ubuntu 20.04 server-system too.
Please help me with some questions:
Where can i find this “main.conf” file ? , i didnt find it in the spamassassin, amavis and in postfix directories.
what is and where can i find “FSC_CENTER_DIR” ?
where can i find “system.filter” ?
what is the name of the called php-script ?
sorry for my stupid questions, i am no professional !
thanks, danke
Gerald

2
0
Would love your thoughts, please comment.x