Configure Sendmail For HELO Check

What you basically want to do is check the HELO/EHLO name in the check_mail ruleset. This is done by putting macro $s into the workspace and then testing for a dot. This by itself is quite simple. You would add the rules to the Local_check_mail ruleset:

SLocal_check_mail
# replace workspace with $&s ($& is delayed evaluation of the macro)
R$* $: $&s
# if $s is null (not set) then we are not in typical SMTP mode, exit
# nothing -> exit ruleset with OKSOFAR
R$@ $@ OKSOFAR: macro S not set
# test if $s has a dot in it, if so exit ruleset
# something . something -> exit ruleset with OKSOFAR
R $+ . $+ $@ OKSOFAR: macro S has a dot
# Finally if we are still here, then $s was set and did not have a dot
# something -> exit calling the error mailer
R $+ $#error $@ 5.5.0 $:550: Bogus HELO/EHLO name

Now if you wanted to be a bit more careful about alternate SMTP modes, then you would copy in the first two blocks of rules for deferred and authenticated modes.

The second thing you asked about is how to use this test to label the message rather than rejecting the message. This is something I cover in my "Managing Internet Mail" class. The basic rules are the same, except for the last rule. In that rule you set a macro rather than call the error mailer. You also need some glue to generate a custom header. First the glue:

# define the macro map for setting macro values while in the rulesets
Kmacro macro
# define a custom header to indicate a bad HELO/EHLO name
H?${BadHELO}?X-bad-HELO-cmd: Host name in HELO/EHLO did not have any dots
# Add macro ${BadHELO} to class {persistentMacros} so that it is
# stored in the qf file between queue runs (no $ sign before macro name)
C{persistentMacros} {BadHELO}

Next modify the final rule to set the macro:

Local_check_mail
.
.
.
# Finally if we are still here, then $s was set and did not have a dot
# something -> exit setting macro ${BadHELO}
R $+ $@ $(macro {BadHELO} $@ TRUE $) TRUE $1

If you were adding these rules with M4, then I would recommend you put them in a HACK() by creating a file .../hack/badhelo.m4

The glue part you would include with the M4 tag:

LOCAL_CONFIG

and the rules you would add with:

LOCAL_RULESETS

You would then include the hack in your host.mc file with:

HACK(badhelo)