GW Tester: testing S/P-GW without pains

TL;DR

• I released GW Tester: a set of tools to test S/P-GW as easy as possible in modern way.
  • pseudo MME that manages sessions on S/P-GW and communicate with pseudo eNB with gRPC.
  • pseudo eNB that creates GTP tunnels with S-GW and can inject traffic from external node or generate it by itself.
• Quick demo in this post and setup instructions in the repo are available.

Mobile core network is one of the most complicated systems that is designed to work perfectly when every node works in the manner compliant with the specifications, that is, if we want to test a single node(let’s say it’s P-GW) but only a small part of other nodes(eNB/MME/S-GW etc.) are not mature enough, we may not even be able to finish an attach procedure for a single subscriber. It requires all the interfaces up an running, including the ones that are not important for what we want to test in a certain situation.
Also, using the open source projects like OAI, srsLTE, etc. just for testing a node is so much a burden that I want to avoid (personally I’m not interested in mastering how they work specifically... It’s really painful for me to get used to such kind of monolithic large projects).

One of the biggest reason why I’ve been working on implementing minimal protocol stacks used in mobile core network(list of my projects) is to make this situation better, and to make it easier for developers to join telco developments.

And, fortunately, I’ve got an encouraging comrade recently, who is thinking about realizing similar things: CNCF's CNF Testbed initiative. To make telco infrastructure work in “cloud-native" way, it’s important to keep components minimal. Since I thought that was the exact field to which I could contribute with my (lonely) projects and it's good for attracting more developers to telco, I started to working with them on implementing cloud-native testing tools and functional products.

Our approach

In that context, I discussed what should be useful for their testbed, and created a set of tools named GW Tester as a prototype product within go-gtp project, which consists of pseudo eNB and MME, implemented specifically to let S/P-GW work.

Normally in LTE, before creating sessions on S/P-GW, we need to perform at least;

• Setting up bidirectional S1AP connection between eNB and MME
• Authenticating UE by HSS with complicated procedure/algorithms
• Exchange some information to establish bearers

Yes, these are the most vital parts for LTE to work secure and easy for subscribers and operators, but sometimes we want to skip them for testing and demos. Since there has not been such tools so far as long as I know, I created a set of tools that;

1. creates sessions based on static configuration written in yaml, without any authentication
2. lets users easily inject U-Plane traffic from inside/outside of the pseudo eNB

I’ll describe how each functionality works with my thoughts behind that. The project has some documentations in the repo, which might be more friendly for engineers/developers than reading this post :)

How it works

1. Session Creation

S1-MME interface(S1AP and NAS protocols) is substituted by gRPC, and that is kept as simple as possible for creating sessions on S/P-GW. It’s unidirectional, and thus no connection state is maintained between eNB and MME, which means that MME cannot initiate the packet exchange with eNB. It's omitted as it doesn't actually matter for testing S/P-GW.
eNB sends the information that is required to create session all at once in the first protobuf message, and then MME sends real GTPv2 messages(Create Session/Modify Bearer) to S-GW. After successful creation of the session on S/P-GW(=Cause in response from S-GW indicates success), MME tells eNB the IP address of S-GW and outgoing TEID assigned to the session that eNB should use in the GTP-U tunnel on S1-U I/F.

In this procedure, however, two important factors are omitted in the current implementation: gateway selection(usually done by DNS lookup with APN) and IP address assignment to UE(by P-GW/AAA). I’m still in the middle of consideration of how it should be, so the behaviors might change in the future.

Anyway, while this procedure works, S/P-GW may think that there are real eNB/MME working right and get themselves ready for forwarding GTP-U packets.

2. U-Plane Data Injection

There are currently two ways to inject the traffic into U-Plane, and that can be configured for each subscriber(=per source IP, network interface).

External client

This mode enables eNB to forward traffic coming to the specified network interface to GTP device where Kernel GTP-U encapsulates the traffic and forward to the peer.

So users can use any machines(or containers) that have specified IP address and are reachable to eNB as a UE.

Note that both client and server should have appropriate routing entry to forward packets to eNB/P-GW like the following.

On client:

$ sudo ip r add <network the servers belong> via <eNB's IP on eUu side I/F> dev ...

On server:

$ sudo ip r add <client's IP or network segment> via <P-GW's IP on SGi side I/F> dev ...

built-in traffic generator

This mode is more demo-ish. eNB adds the specified IP address to specified network interface, and sends packets towards a specified destination via GTP tunnel. It can be used to show packets going through tunnel without any additional machine, but the traffic is not under users’ control.

Currently, only the plain HTTP is supported in this mode, and ICMP is still under consideration(Let me know if there’s good library to send ping and get results with Go).

In this mode routing on server is required, like the following.

$ sudo ip r add <IP added to eNB> via <P-GW's IP on SGi side I/F> dev ...

OAM

Along with the main functionalities above, GW Tester has some features convenient for operators/administrators.

YAML-based flexible config

All the parameters to make it work are given by a single YAML file, to be easier for operators to update configuration in modern way with orchestration tools.

Also, as other tools does, it reloads config when receiving SIGHUP without restarting(not disconnecting UEs).

Instrumentation with Prometheus

Some metrics are exposed by giving the IP/Port to prom_addr param in YAML config.
The metrics currently implemented are;

• number of sessions/bearers
• number of the C-Plane message sent/received (per message type)

Metrics are very useful and sometimes we think it as a good tool for tracing activity, like count of the session labeled with IMSI etc, but we shouldn't. As the Prometheus official doc says, with these kinds of lebel it can easily be excessive from the cardinality point of view.

Please let me know if you have any idea about other kind of metrics :)

Quick Demo

Do you want to see it’s working? Setup instructions are available in README. Here are some logs and screenshots for those who just want to have a quick look.

Network configuration

Here’s a diagram on the top of this post again with the IP addresses I assigned.

• Every node works on Docker container created with Ubuntu image.
• Network segments are created as Docker overlay internal network.
• Routing entries of external UE and Server are added manually beforehand.
• Small HTTP Server runs on Server with python -m http.server 80.

You can find docker network configuration commands and YAML files for each node in this Gist.
Unfortunately there are no Dockerfiles because I've configured everything manually. I'll upload them when it's ready (some day) :D

Outcome

Here’s a PCAP that is captured on S-GW.

In that, you can see MME/S-GW/P-GW exchanging GTPv2-C packets to establish sessions and tunnels.

And, GTPv1-U encapsulated packets between UE and Server follows.

You can also see the terminal output in GIF format. I don’t put it inline here because it’s too large.

Got interested?

Thank you for reading. If you’re interested in this project, give me feedback on GitHub issues or comments. As Gophers can see that needs more works to be as “a product”, so I’d be very happy with your contributions, even a small parts.

Let’s get in touch with us to make it better together. Hope my product helps someone struggling with mobile networking :)

Find us on Twitter!

• Me(personal): @wmnskdmms
• CNCF CNF Testbed: @cnftestbed
• CNCF NSM: @nservicemesh