You can run a mail server on a VPS. Before deciding to, it is worth being clear that mail is the hardest common service to operate well, and the difficulty is almost entirely about deliverability rather than about installing software.
The honest recommendation first
For most people, use a hosted mail provider and point the domain's MX records at it. Mail then arrives, and the several days of work below become unnecessary.
Run your own when you have a specific reason: volume that makes hosted mail expensive, a regulatory requirement, or an application that needs local delivery. "It comes free with the server" is not one, because the cost is not the software.
Why deliverability is the hard part
Installing a mail server takes an afternoon. Getting large providers to accept its mail takes considerably longer, and nothing warns you when it is not working: messages simply do not arrive.
Four things are required before any of it works.
A clean IP address. VPS addresses are frequently reused, and yours may carry a history from a previous occupant. Check it against the major blocklists before building anything on it. If it is listed, deal with that first. Nothing else matters until it is resolved.
A correct reverse DNS record for the address, matching your mail server's hostname. Many receiving servers reject mail from an address with no matching reverse record, and only your provider can set it.
SPF, DKIM and DMARC published. Without all three, major providers treat your mail as suspect regardless of its content. For publishing them, see understanding SPF, DKIM and DMARC.
Reputation, which takes time. A new address is untrusted until it has sent legitimate mail for a while. Sending a large volume immediately is the fastest way to be treated as a spam source.
Port 25 may be blocked
Many providers block outbound port 25 on new servers to limit spam, and it is often unblocked only on request after review.
Check this before planning anything, because a mail server that cannot open outbound connections on 25 cannot deliver to anyone.
If you do run one
The pieces are an SMTP server for sending and receiving, an IMAP server so clients can read mail, spam filtering, and a certificate so connections are encrypted.
Two things are worth being disciplined about.
Never run an open relay. A server that accepts mail from anyone and forwards it is found by automated scanning within days and used for spam. The address is then blocklisted and your legitimate mail stops arriving too.
Rate-limit outbound mail. A compromised application on the server can send thousands of messages before anyone notices, with the same result.
Application mail is a smaller problem
If you only need the server to send: password resets, order confirmations, contact forms. You do not need a mail server at all.
Configure the application to send through an authenticated SMTP service. That gets you deliverability handled by people who do it full time, with none of the receiving-side work, and it is the right answer far more often than running a full mail server is.
Monitoring, if you run one
Watch the outbound queue and check the address against blocklists periodically. A queue that is growing means delivery is failing somewhere, and finding that out from the queue is much better than finding out from a customer who never received an invoice.
If mail from the server stops arriving anywhere, What to Do When Your IP or Domain Is Blocklisted deals with finding the cause before requesting removal.
Test whether outbound mail can leave at all
Before configuring anything, establish whether the provider permits direct mail delivery, since many do not.
nc -zv -w 5 gmail-smtp-in.l.google.com 25 2>&1 | tail -1 nc -zv -w 5 alt1.gmail-smtp-in.l.google.com 25 2>&1 | tail -1 curl -s ifconfig.me; echo
A refused or timed out connection on that port means outbound mail is blocked at the network, and no configuration on the machine changes it.
That is the common case on virtual servers, and it is a policy rather than a fault. The answer is a relay service, which is also the better arrangement for delivery regardless.
Check the address you were given
An address carries a sending history, and a new machine can arrive with a reputation somebody else created.
dig -x "$(curl -s ifconfig.me)" +short
dig +short "$(curl -s ifconfig.me | awk -F. '{print $4"."$3"."$2"."$1}').zen.spamhaus.org" 2>/dev/null
A response from the second query means the address is listed. That is worth knowing before building anything on it, since a listed address delivers nothing regardless of how the server is configured.
Ask for a different address rather than attempting to clear a listing you did not cause. That request is routine and considerably faster than the alternative.
Send through something built for it
The honest recommendation is not to run a mail server, and the practical arrangement is to send through a service that specialises in it.
openssl s_client -connect smtp.provider.example:587 -starttls smtp </dev/null 2>/dev/null | grep -i auth dig example.com TXT +short | grep -i spf
Confirm the relay's own record is included in your authorisation, or every message it sends on your behalf fails the check that receiving systems apply.
Keep the credentials outside the application's code, and use a sending address that exists and can receive replies. A message from an address that bounces is treated with suspicion by every receiving system.
Watch the queue rather than assuming delivery
A mail server that accepts messages and cannot deliver them keeps them, and the only sign is a queue that grows. Read the queue length regularly and treat any figure that is not returning to zero as a fault rather than a backlog. Messages sitting for days are eventually returned to the sender, which means the failure becomes visible to your customers before it becomes visible to you. The queue is also disk, so an undelivered backlog on a small machine becomes a full disk and a second, larger problem.