Standard encoding leans on formats every network stack already understands: base64, hex, MIME, sometimes plain ASCII substitution. BADNEWS is the textbook case: it encrypts its C2 payload, converts the ciphertext to hexadecimal, then base64-encodes that hex string before it ever touches the wire. The result rides inside something mundane, an HTTP header, a User-Agent string, a cookie value, or a DNS query label, so a packet capture shows what looks like a normal web request rather than an obvious blob of ciphertext. YamaBot, attributed to Lazarus, base64-encodes its entire User-Agent string and stores RC4-encrypted, base64-encoded tasking inside a cookie named captcha_session, betting that analysts skim past cookie values the way they skim past User-Agent strings.
Non-standard encoding drops the shared alphabet and substitutes a scheme only the malware and its operator know, XOR against a hardcoded key before base64, a custom character-substitution table, or a passkey-gated format like Velvet Ant's, where commands only decode correctly if the responding implant already holds the right key. This buys resistance to generic decoders and signature matching built around known encodings, at the cost of needing the malware's own code (or a reverse engineer) to unwrap it.
Either way, encoding is a wrapper, not a lock. Base64 and hex are trivially reversible by anyone who notices the pattern, which is why detection here targets encoding artifacts (fixed prefixes, padding characters, alphabet composition, length) rather than trying to break anything cryptographic. The traffic frequently is not encrypted at the encoding layer at all, TLS handles confidentiality in transit and the encoding exists purely to make the payload look like ordinary text to a human or a naive content filter.