Replacing a Licensed SS7 Stack With Go: What the Wire Taught Me

Over the last few weeks I wrote a small SIGTRAN stack in Go (SCTP, M3UA, SCCP, TCAP) to replace a commercial SS7 stack under one of our signaling apps. Writing the protocol code was the easy part. Most of what I learned came from the Linux kernel and from a second host that shares our point code.

Why build one at all?

If you come from the cloud-native side, SS7 can look like archaeology. It still carries a lot of the mobile core, though: location updates, authentication vectors, SMS routing, and CAMEL call control and charging. If your network talks to an HLR, an MSC or an SMSC, it probably speaks MAP over TCAP over SCCP, and these days that runs on IP using SIGTRAN.

Our app is a signaling gateway. It talked to a licensed vendor stack through a native addon. That worked, but it meant a license, vendor RPMs and a native build on every deploy. I also wanted the stack in its own process, so the app could restart without taking the SCTP associations down with it.

Then I looked at what we actually used, and it was a short list:

• one flavor of M3UA, as an ASP, with no routing contexts;
• one SCCP variant, connectionless only;
• part of TCAP (we never send TC-U-REJECT, result-not-last or TC-UNI);
• MAP and CAP on four subsystems: HLR, MSC, EIR and CAP (SSNs 6, 8, 9 and 146).

Writing a general-purpose SS7 stack would take years. Writing one that does exactly this, and clearly refuses everything else, is a project you can finish. So I wrote it.

A 2-minute SIGTRAN primer

If you have never touched SS7, here is the stack from bottom to top:

+------------------------------------------------------+
| MAP / CAP    the "application": UpdateLocation,      |
|              SendAuthenticationInfo, InitialDP, ...  |
+------------------------------------------------------+
| TCAP         dialogues (Begin/Continue/End/Abort)    |
|              and components (Invoke/Result/Error)    |
+------------------------------------------------------+
| SCCP         addressing: point code, SSN, global     |
|              title (GT), and returned messages       |
+------------------------------------------------------+
| M3UA         carries MTP3 user data over IP;         |
|              ASP state toward the signaling gateway  |
+------------------------------------------------------+
| SCTP         reliable, message-based, multihomed     |
|              transport (the TCP of telecom)          |
+------------------------------------------------------+

• SCTP is a transport like TCP, but it sends whole messages, and one association can use several IP paths at once (multihoming). If one path dies, traffic moves to the other.
• M3UA (RFC 4666) lets an IP host act like an SS7 node. Our host is an ASP (application server process) and the STP is the signaling gateway (SG). The ASP has to send ASP Up and then ASP Active before the SG sends it any traffic.
• SCCP adds addressing on top of point codes: subsystem numbers and global titles. The STP does global title translation for us.
• TCAP is the transaction layer. A dialogue has an ID at each end, and each message carries components, which are basically RPC calls and replies.
• MAP and CAP are the operations themselves, encoded in ASN.1.

Something that confuses everyone at first: SS7 has regional variants (ANSI and ITU, mainly), and one network can mix them, with one variant for MTP and SCCP and another for TCAP. Nobody tells you which mix you have. You find it in the config files and the captures.

How the daemon is built

The new stack is one Go daemon per host. The app talks to it over a Unix socket, through a small adapter that has the same functions and the same events as the old addon. The MAP and CAP handlers did not change at all, which made the switch much less scary.

          app (MAP/CAP handlers, unchanged)
                     |
                     |  Unix socket, one frame per call or event
                     v
+-----------------------------------------------------+
| daemon                                              |
|   app interface   addon calls/events <-> TCAP       |
|   TCAP            dialogues, components, timers     |
|   SCCP routing    primary/secondary STP, SST, UDTS  |
|   M3UA            one ASP per SG process            |
|   SCTP            Linux kernel SCTP, multihomed     |
|   peer link       TCP to the other host (more below)|
+-----------------------------------------------------+
                     |  SCTP, two paths per association
                     v
                 STP (signaling gateway)

Some design choices that paid off:

The protocol packages do no I/O and keep no clock. SCCP, M3UA and TCAP take bytes plus the current time, and give back bytes and events. They never touch a socket and never call time.Now(). Because of that I could unit test them, fuzz them and replay captured traffic through them without any network. The shape is roughly this (simplified, not the real API):

1
2
3
4
5
6

// The stack never reads the clock itself; the daemon passes it in.
type Stack interface {
    Receive(now time.Time, msg []byte) []Event // from the network
    Send(now time.Time, req Request) ([]byte, error)
    Tick(now time.Time) []Event                 // fire due timers
}

One goroutine owns the state. The daemon has a single loop that owns the TCAP node and the routing state, and the SCTP links and the app connection send it events over channels. A remote peer decides how much traffic hits that state, and I did not want locks a peer could hold open. The race detector was a lot quieter for it.

Each association has its own send queue and its own sender goroutine, so a peer that stops reading cannot stall the main loop. When a link’s queue is full, the send returns “busy” and the router tries another link.

The only dependency is golang.org/x/sys, for the SCTP socket calls. All in, it is about 12k lines of Go, with a bit more than that in tests.

Lesson 1: let the captures define “correct”

The goal was wire parity with the old stack: the same STPs, from the same local addresses and ports, with nothing changed on the STP side. The specs (Q.773/Q.774 for TCAP, RFC 4666 for M3UA, and the SCCP spec for your variant) tell you what is allowed. Captures tell you what your network actually does.

I wrote a small tool that decodes every M3UA message in a pcap, re-encodes it, and fails if the bytes differ. Every message in our lab captures comes back byte for byte. I anonymized those messages and kept them as golden test vectors, and the tests replay the captured dialogues through the TCAP stack.

The captures also showed things I would never have got from a spec:

• DATA always goes on SCTP stream 1, never stream 0.
• The SCCP SLS (signaling link selection) changes on every message, even inside one dialogue.
• The last bytes of the transaction IDs seemed to say which host owned the dialogue.

That last one ended up mattering a lot (lesson 4).

Lesson 2: Linux kernel SCTP works, but check the defaults

I used the kernel’s SCTP through golang.org/x/sys/unix, with one-to-one sockets. It works well. Its defaults, though, are tuned for the open internet, and a signaling link is a different animal. The max RTO defaults to 60 seconds and the path heartbeat to 30 seconds, where you want sub-second retransmission and a dead path noticed in seconds. And delayed SACK is on.

The delayed SACK one cost me the most head-scratching. In the first lab run with real traffic, about a third of the inbound messages arrived twice. The STP resent each one on the other path roughly 120 ms after the first copy, because our ack was sitting in the kernel’s SACK delay. The fix is one socket option: SCTP_DELAYED_SACK with a delay of 0 and a frequency of 1, so every packet is acked at once. Since then I set every SCTP timer explicitly and trust none of the defaults.

Two more surprises at the host level.

Multihoming needs host routes. Kernel SCTP takes the source address from the routing table, and our two interfaces were on the same subnet, so both paths left through the first interface. We had “two paths” on one NIC. One host route per STP address, each out of its own interface, fixed it:

1
2
3
4
5

# Example only, with documentation addresses.
# Path 1: local 192.0.2.10 on eth0 talks to STP address 198.51.100.1
ip route add 198.51.100.1/32 dev eth0 src 192.0.2.10
# Path 2: local 192.0.2.11 on eth1 talks to STP address 198.51.100.2
ip route add 198.51.100.2/32 dev eth1 src 192.0.2.11

Kernel SCTP and a user-space SCTP stack cannot live on the same host. The old vendor stack ran its own SCTP in user space. As soon as the sctp kernel module is loaded, the kernel answers SCTP packets meant for that stack with ABORT. The module also never unloads, so switching from one to the other means a reboot. Now each side refuses to start while the other is active.

Lesson 3: “connectionless” still has ordering

SCCP class 0 is connectionless and promises no ordering. The old stack rotated the SLS on every message, so one dialogue’s messages could take different links, and that was fine most of the time.

Then a CAMEL call failed. The switch opens the dialogue with an InitialDP in protocol class 1, which means “keep these in order”. We answered with two messages, a RequestReportBCSMEvent and then a Continue. Each got a new SLS, they took different paths, the Continue arrived first, and the switch aborted the dialogue.

Captures of the old stack confirmed the rule: when the peer opens a dialogue in class 1, answer in class 1, and use one SLS for every message in that dialogue. We now derive that SLS from the dialogue ID. It is a one-line rule once you know it. Before you know it, it is an afternoon of staring at captures.

Lesson 4: two hosts, one point code

We run two hosts that together act as one SS7 node, with one point code and one global title. The STP routes by point code. It knows nothing about transactions, so when host A opens a dialogue, the Continue or End coming back can land on host B.

This is not a rare edge case. A large share of the answers a host receives belong to dialogues on the other host. The old stack spanned both hosts and dealt with this out of sight. A daemon per host has to deal with it in the open.

First, the owner goes into the transaction ID. The top bit of every 4-byte ID says which host owns it, then comes a generation drawn at random, then a slot. A late message for a reused slot does not reach the new dialogue, and a peer cannot guess a live ID by counting.

Second, the two daemons have a peer link: a plain TCP connection with a small frame format. A message for the other host’s ID is forwarded as it came, and the owner answers on its own STP links. Nothing goes back over the peer link.

Third, the subsystems belong to the node. If one host’s app restarts, that host must not send SSP (subsystem prohibited), or the STP stops sending that subsystem’s traffic to both hosts. A subsystem stays in service while either host’s app is attached, and a host without its app hands new dialogues to the other one.

The cleaner fix is a point code and a GT per host, so answers come back to the right host in the first place. That needs changes on the STP side, though, so for now the peer link is what we run.

Lesson 5: review like you do not trust the tests

The protocol layers went through several review rounds. The most useful ones started cold. The reviewer had never seen the code, the design or the earlier notes, and had only the specs and the tests to go on. Each one worked on a copy, so a finding came with a failing test and we did not have to argue about it.

I now do two things on every change.

1. Break the code a test guards and check that the test goes red. One address-encoding test looked thorough, with six cases. A missing field made all six fail on the very first check, so you could delete any of the six checks it claimed to cover and the suite stayed green. Another assertion, “must reject class 2”, accepted any error at all, so it passed even with the class check removed.
2. For every state change, ask what happens if the answer arrives late. One finding: if the daemon asked to go inactive while its ASP Active request was still unanswered, the late ASP Active Ack was ignored. The STP then held the ASP as active while the ASP held itself inactive, and every DATA got an error back. A SIGTERM right after connect was enough to trigger it. Now a late Ack gets answered with ASP Inactive.

Every fix from review got a test, and every one of those tests was checked to fail on the old code. The codecs also have fuzz targets that decode, re-encode and compare, plus one that throws random input at a live TCAP stack.

Where it stands

In the lab the new daemon carries real MAP and CAP traffic end to end. Next are load and soak tests, with link and process failures thrown in.

The specs were the easy part. The hard parts were the kernel defaults and the things the old stack did quietly that nobody had written down. Most of those only show up in a capture.

If you are working on something similar, or have stories about SCTP on Linux, I would like to hear them.

Take care and stay safe! ;)

Useful Links

1. RFC 4666: M3UA
2. RFC 9260: SCTP
3. RFC 6458: Sockets API Extensions for SCTP (where SCTP_DELAYED_SACK and friends are defined)
4. ITU-T Q.771 to Q.775: TCAP
5. 3GPP TS 29.002: MAP
6. Linux kernel SCTP documentation