How to stop Courier from sending Cannot display Unicode content on pop3

last week we upgraded our webserver from buster to bullseye. within this, our mailserver courier was updated from version 0.76.3-5 to 1.0.16. we downgraded a year or more ago when we did upgrade from stretch to buster, cause we was running into the same problem after the uprade: mails, that were retrieved via pop3 always was „corrupted“ cause courier did send a warning message with the subject cannot display unicode and following content:

This E-mail message was determined to be Unicode-formatted
but your E-mail reader does not support Unicode E-mail.

Please use an E-mail reader that supports POP3 with UTF-8
(see https://tools.ietf.org/html/rfc6856.html).

This can also happen when the sender's E-mail program does not
correctly format the sent message.

The original message is included as a separate attachment
so that it can be downloaded manually

after the try to downgrade again to 0.76 which leaded to dependencie problems cause we use mysql for authentication and i didnt wanted to downgrade all the mysql-stuff, studieing the changelogs of courier, change all the last customers from pop3 to imap, cause i couldnt find another solution, i decided to drop Sam, who is one of the main developer of courier, a mail. all the fixes he did relatet to this unicode-alerts were made only for IMAP. he confirmed in his mail, that there were no improvements or changes for pop3 in the last years, and he will not implement some, cause mailclients should not send binary garbage. after some mail-exchanges, he told me that inside the junk-header, the content-preview contains the binary garbage, that causes this alerts. this content-preview is generated by spamassassin. so i stopped spamassassin, send a test mail, and voilat: the problem disappeared! so if one of you will run into the same problem, here is the solution: temporarly i disabeld the spam-report (commenting add_header = X-New-Spam-Report: $acl_m_spam_report) in our acl.spamassassin.conf, that we use with exim.

i will try to fix the broken character-transmitting inside the report when i have more time. will update this post than!

So, here is the update

Following Solution:

in our main.conf

#UTF8 Domain PERL Stuf
allow_utf8_domains = true
perl_startup = do 'FSC_CENTER_DIR/exim/perl/exim.pl'
perl_at_start = true

in this exim.pl

sub header_encode {
	$headerName = shift;
	$headerValue = shift;
	
	$dir = dirname(__FILE__);
#	$eHeaderName = encode_base64($headerName);
#	$eHeaderValue = encode_base64($headerValue);

		#open ( DEBUG, ">>/tmp/header_encode_debug.txt" );
                #print DEBUG "PERL IN: " . $header . "\n";
                #print DEBUG "PERL EXEC: /usr/bin/php $dir/exim.php \"header_encode\" $eHeaderName $eHeaderValue\n";
                #close ( DEBUG );

	$newheader = `/usr/bin/php $dir/exim.php "header_encode" '$headerName' '$headerValue'`;
	
	return ($newheader);
}

…so cause we cant code PERL we just call a php script to do the final encoding.

<?php
ob_start();
error_reporting(E_ALL);

$returnCode = 1;

$todo = isset($_SERVER['argv'][1]) ? $_SERVER['argv'][1] : null;

exec('echo ' . escapeshellarg('PHP TODO: ' . escapeshellarg($todo)) . ' >> /tmp/header_encode_debug.txt');
exec('echo ' . escapeshellarg('PHP IN: ' . escapeshellarg(json_encode($_SERVER['argv']))) . ' >> /tmp/header_encode_debug.txt');


if($todo == 'puny') {
	include(__DIR__.'/IDNAConvert.class.php');

	$address = $_SERVER['argv'][2];
	//$local = substr($address, 0, strpos($address, '@'));
	$domain = $address; //strtolower(substr($address, strpos($address, '@') + 1));

	$returnCode = 0;

	$convert = new IDNAConvert();
	$punycode = $convert->encode($domain);
	if($punycode == null) {
		$address = '';
		$returnCode = 2;
	}
	else {
		$address = $punycode; //"$local@$punycode";
		$returnCode = 0;

		if(!preg_match('/^[a-z0-9\.-]+$/i', $punycode)) {
			$address = '';
			$returnCode = 3;
		}
	}

	echo $address;

	//exec('echo "PHP IN: \"' . $address . '\"" >> /tmp/punycode_debug.txt');
	//exec('echo "PHP PUNY: \"' . $domain . '\" -> \"' . $punycode . '\"" >> /tmp/punycode_debug.txt');
	//exec('echo "PHP RES: \"' . $address . '\" -> \"' . ob_get_contents() . '\"" >> /tmp/punycode_debug.txt');
}
if($todo == 'header_encode') {
	require_once('phar://'.__DIR__.'/FISBMail.2.phar/index.php');

	$headerName = $_SERVER['argv'][2];

	$headerValue = base64_decode($_SERVER['argv'][3]);

	$ho = BMailHeader::create($headerName, $headerValue);

		#file_put_contents("/tmp/header_encode_debug.txt", "PHP IN:\n$headerName: $headerValue\n", FILE_APPEND);

	$result = $ho->getHeaderLinesData();

		#file_put_contents("/tmp/header_encode_debug.txt", "PHP RESULT:\n$result\n", FILE_APPEND);

	$result = "$headerName: " . trim($result);

		#file_put_contents("/tmp/header_encode_debug.txt", "PHP RESULT:\n$result\n", FILE_APPEND);

	echo base64_encode(str_replace("\r\n", "\n", $result));

	$returnCode = 0;
}

die($returnCode);

and finally use it, under our system.filter we added:

if $acl_m_addspamreportheader is not "" then
headers remove X-Spam-Report
headers add ${base64d:${perl{header_encode}{X-Spam-Report}{${base64:$acl_m_addspamreportheader}}}}
endif

thats all, hope this helps someone 🙂