AndroSSL: A Platform to Test Android Applications

Connection Security

François Gagnon, Marc-Antoine Ferland, Marc-Antoine Fortier,

Simon Desloges, Jonathan Ouellet, and Catherine Boileau

Cybersecurity Research Lab

Cégep de Sainte-Foy, Québec, Canada,
[email protected]
http://www.cegep-ste-foy.qc.ca/cybersecurity

Abstract. Developing secure mobile applications is not an easy task; especially

when dealing with SSL/TLS since very few developers possess experience with
those protocols. This paper presents AndroSSL, an automated platform to as-
sess the security of (SSL/TLS) connections established by Android applications.
AndroSSL assists mobile application developers by testing their applications for
man-in-the-middle attacks, and, successful, pinpoints the reason why the appli-
cation is vulnerable.

Keywords: Privacy, SSL, MitM, Android, Test-Bed, Automated Experiment.

1 Introduction
Mobile application developers are facing a new and difficult security challenge. While
traditional web applications, common in the desktop world, rely on web browsers to
manage secure communications, each mobile application must deal with this element
on its own. Establishing a secure channel using the SSL/TLS protocol [3] requires the
client to check the validity of the SSL1 certificate received from the server. An appli-
cation accepting an invalid certificate would allow an attacker to impersonate the real
server through a man-in-the-middle (MitM) attack.
The last decade has shown that validating an SSL certificate is a difficult and error-
prone task. Even big players (e.g., web browser developers) have a hard time getting
it right (see2 CVE-2008-4989, CVE-2009-1358, CVE-2009-2510, CVE-2009-3046,
CVE-2010-1378, CVE-2014-1266). It would be unrealistic to believe all (even most)
mobile application developers will rise to the task easily; especially since a lot of mobile
applications are developed by non-expert programmers (much less security specialists).
AndroSSL is a framework aiming to help mobile developers test their applications
against connection security flaws. It relies on virtualization to provide a low cost and
highly automated platform. Moreover, by offering a wide range of tests (in several dif-
ferent contexts), it is meant to provide detailed information regarding vulnerabilities in
the certificate validation process.
1 The terms SSL and TLS are used interchangeably throughout this paper.
2 http://www.cve.mitre.org/
 The paper is structured as follows: Section 2 provides an overview of AndroSSL,
while Section 3 details the methodology used to mount MitM attacks. Section 4 dis-
cusses related work. Finally, Section 5 concludes with a summary of our findings and
an opening on upcoming future work.

2 Experiment Test-Bed

The objective of AndroSSL is to perform automated SSL MitM attacks against An-
droid applications. When a MitM attack succeeds, it stands as proof that the application
is vulnerable. Automation makes AndroSSL interesting for non-security experts (e.g.,
mobile app developers), since AndroSSL users do not have to worry about the inner
working of security protocols.
To automated its process, AndroSSL uses the AVP test-bed which relies heavily on
virtualization to accomplish its mission. Android virtual devices (AVD) running on the
Google QEMU emulator are used to host the applications to be tested, while standard
virtual machines running in VMWare Workstation are used to host the required custom
servers (e.g., DNS, Gateway, Fake SSL server). AndroSSL will run on any machine
capable of running the Android Development Kit.
When testing the security of an application, AndroSSL will go through eight steps:

1. Load a scenario describing the experiment.

2. Configure the network services needed (e.g., DNS, Fake SSL server).
3. Start an AVD.
4. Install the targeted Android application and launch it.
5. Start recording different aspects of the experiment (e.g., network traffic, Android
internal log (logcat), screenshots of the AVD).
6. Interact with the application to trigger a legitimate login attempt.
7. Serve a fake3 SSL certificate to the application.
8. Analyze experiment logs to determine if a security flaw was successfully exploited.

The experiment specifications (scenario) is given to the system through an XML

file. The scenario indicates which AVDs to use for the experiment, what information is
to be recorded (e.g., network traffic, Logcat) and what actions are to be performed by
the AVDs.
During the execution phase, AVP manipulates AVDs through the instrumentation
of various Android virtualization technologies. It executes the AVDs with their applica-
tions and then collects experiment data. AVP supports a wide range of actions and data
collection capabilities. Regarding AndroSSL, the most important actions are: start/stop
AVDs4 and install/start applications. AVP supports several data gathering capabilities;
those leveraged by AndroSSL are: network traffic recording and Android logcat record-
ing.
Once the experiment is over, post-analysis of the log files can be performed, see
Section 3.2.
3 Meaning a certificate that should be considered invalid from the application’s point of view.
4 With the possibility to use snapshots.
2.1 User Simulation
The most challenging requirement of AndroSSL towards its test-bed is the need for
proper user-simulation to lead an application to a specific state in order to trigger re-
mote (possibly secure) connections. Our approach has been to manually pre-record
(only once for each app) the sequence of actions leading the application to initiate a
secure connection, then replay this action sequence automatically (as many times as
needed) in AndroSSL experiments. Although this introduces manual intervention, it
does not deviate from the objective of AndroSSL to help developpers test their apps
automatically. Indeed, not every developer has the knowledge to mount a MitM attack
or to analyze and interpret the network traffic and logs, but, it will be easy for them to
record their intended action sequence.

3 Methodology
The methodology behind AndroSSL has two components: the various scenarios used
to attack the tested applications and the analysis of collected data to determine whether
the attack was successful or not. Since some results will be discussed, a description of
the experiment dataset comes first.

3.1 Dataset
AndroSSL has been used to test 90 Android applications against 6 different attack sce-
narios. All these applications required Internet access and were downloaded from the
top 500 most popular of each categories on the Google Play Store.

3.2 Analysis
Three components are considered during the automated analysis of a run (that is, one
application tested against one attack scenario): SSL server logs, network traffic and
Logcat.
When the SSL handshake is completed with a fake server, the success for the MitM
is confirmed. Hence, the server logs are a good source of information.
Network traffic is analyzed to find the plain password or a standard unsalted hash.
This method allows to applications not using SSL to exchange credentials. Custom or
salted hashes are more difficult to handle. In these cases, a heuristic search for keywords
could be used, but manual confirmation would be required.
Initially, logcat was not intended to be part of our result analysis. During the devel-
opment phase, a search query was run by mistake on the logcat output file and returned
surprising results. The password we used was broadcasted inside the AVD for a number
of apps (10 out of 90). It turns out that those apps were relying on an API to query the
server. The API will (or won’t) use SSL properly to send the credentials to the server;
this is also tested by AndroSSL. The broadcast of credentials inside an Android device
is problematic as another installed application could manage to receive those broad-
casts5 (e.g., the CatLog Logcat Reader App6 ).
5 Since Android 4.3, root privileges are required to access logcat.
6 https://play.google.com/store/apps/details?id=com.nolanlawson.logcat
 Fig. 1. Static URL Network Configuration

3.3 Attack Scenarios

The chosen apps have been tested against 6 attack scenarios. These scenarios, as well
as the results, are presented below.

3.3.1 Actual Server The first scenario consists of letting the application connect to
its actual server without any attempts to attack the app. No particular network configu-
ration is required here. No meaningful results were expected. However, analysis of the
network traffic turned out to be quite rich:
– 9 apps send the user’s password in clear to the server. Anyone capturing the network
traffic would catch the credentials.
– 1 app hashes (SHA-256 unsalted) the user’s password before sending it to the server
un-encrypted.

3.3.2 Static URL Two attack scenarios build the certificate using a static hostname.
That is, the hostname is not related to the URL requested by the application. The net-
work configuration schema of Figure 1 is used for these scenarios. The app queries a
fake DNS (step 1) which will redirect the traffic to our fake SSL server (step 2). Every
SSL connection will be served using the same static certificate (step 3). Two test cases
rely on a static certificate.

3.3.2.1 Self-Signed This scenario is the easiest MitM involving SSL to perform for
an attacker: create a self-signed certificate for a random hostname and use that certifi-
cate for a MitM attack. Figure 1 shows the network configuration underlying this attack
scenario. Applications vulnerable to this attack are broken at two levels: chain valida-
tion (the certificate is self-signed) and source validation (the certificate is not for the
expected hostname). 8 apps were vulnerable to this attack: 2 hash the password with
MD5 (unsalted), the other 6 do not hash.

3.3.2.2 Valid This scenario is also easy to perform for an attacker: obtain a valid cer-
tificate for a domain you really own (15 minutes and 15 dollars) and use that certificate
for a MitM attack. Being a static case, the network configuration of Figure 1 is used
again here. Applications vulnerable to this attack are broken at source validation (the
certificate chain is legitimate, but it is not for the expected hostname). The same 8 apps
as in the previous case fell for this attack.
 Fig. 2. Dynamic URL Network Configuration

3.3.3 Dynamic URL Three attack scenarios build the certificate using a dynamic
hostname. That is, the hostname is related to the URL requested by the application.
For the test cases requiring a dynamic certificate (i.e., where the certificate’s hostname
matches the one requested by the application), a more complex network configuration is
deployed, see Figure 2. The DNS server attributes sequential IP addresses by increment-
ing the last byte by one at every request (step 1 in Figure 2). The first request receives IP
100.100.100.1, the second gets 100.100.100.2, and so on. Then, a Gateway (IPTables7 )
will redirect SSL traffic towards the fake SSL server, but to a specific port (steps 2
and 3). For instance, IP 100.100.100.1 would get port 11001, while IP 100.100.100.255
would get 11255; the last three digits of the port match the last byte of the IP. When
the SSL server receives a request on port 11xyz, it can associate this request with the
URL originally requested to the DNS server (port 11xyz implies IP 100.100.100.xyz,
which can then be matched to a URL in the DNS memory). Hence, the server is able to
dynamically construct an SSL certificate with the expected hostname (step 4).

3.3.3.1 Self-Signed This scenario requires generating a self-signed certificate at run-

time for the specific hostname requested by the app. Applications vulnerable to this
attack are broken at chain validation (the certificate is self-signed). 12 applications fell
for the attack, including the 8 discussed in the previous (static url) attacks. The 4 new
vulnerable applications validate the hostname properly but not the chain (none of them
hashes the password). The others were already known to be broken both for hostname
and chain validation.

3.3.3.2 Signed by Self-Signed A self-signed certificate is generated once. Then, for

each request a certificate is generated for the corresponding hostname and signed by the
self-signed. Applications vulnerable to this attack are broken at chain validation. The
same 12 apps as in the previous case fell for this attack.

3.3.3.3 Signed by Real A real certificate is first obtained for a controlled URL (like in
Section 3.3.2.2). Then, for each request a certificate is generated for the corresponding
7 http://ipset.netfilter.org/iptables.man.html
hostname and signed by the real certificate. Applications vulnerable to this attack are
broken at chain validation for a specific check: a real end-user certificate should not have
the authority to sign other certificates (it is neither a root nor an intermediate certificate;
it is a leaf). 12 applications fell for the attack, the same 12 as in the previous scenario.

3.3.4 Other Scenarios Since there are so many subtleties when validating SSL cer-
tificates, other scenarios will be created to test different flaws (e.g., expired certificates).

4 Related Work

[6] performed a manual analysis of popular security libraries and concluded that SSL
certificate validation is completely broken in non-browser software. Their analysis is
not targeted towards Android applications and is fully manual. Hence, this method is
not appropriate to average developers testing their apps.
[4] proposed an automated approach relying on static analysis of Android applica-
tion source code to find potential SSL certificate validation problems. An automated
static analysis of 13 500 Android applications by their Mallodroid tool flagged 1 074
of them as being potentially vulnerable to MitM attacks. Furthermore, manual MitM
attacks were performed on a subset of 100 potentially vulnerably apps. This analysis
confirmed 41 vulnerable to MitM. Although the static analysis is automatic, manual
work by a security expert is needed to avoid false positives (apps flagged as being vul-
nerable while they cannot be exploited). Again, not entirely suitable for a developer
seeking to test his apps.
[1] proposes a technique to automatically test certificate validation in SSL libraries.
They target libraries implementations and not (Android) applications. Having correct
SSL libraries is essential because apps using a broken library risk inheriting its vul-
nerabilities. However, having perfect libraries does not mean no vulnerabilities in ap-
plications as developers can misuse libraries introducing vulnerabilities. Hence testing
applications and SSL libraries for vulnerabilities is complementary.
[7] manually tested 100 Android Apps for MitM attacks. 32 accepted invalid cer-
tificate opening the door to a MitM attack. Their conclusion is that "...[there is a] need
to give developers more effective tools that can help them detect and fix issues before
the app is in production...".
[5] proposes a way to improve the security of mobile applications by changing
development practices related to SSL connections handling. Although this major step
would certainly have a positive outcome, exhaustive testing will always play an impor-
tant role (especially in security sensitive areas where new flaws are to be discovered).
Google released the nogotofail [2] tool to test client side SSL certificate validation.
A few difference with AndroSSL are worth mentioning. nogotofail provides no au-
tomation for user interaction on the client side. The user interaction has to be done (and
repeated) entirely manually. Having no automation for user interaction means nogoto-
fail is not limited to Android applications; it can be used to test any client. nogotofail is
not limited to client-side validation as it includes bugs like Heartbleed, POODLE and
gotofail. It seems that two scenarios are built in nogotofail to test for client-side certifi-
cate validation: one static (Invalid Hostname Certificate [2]) and one dynamic (hinted
in [8]). It is not clear how easily nogotofail can be extended with other scenarios.
[9] proposed an approach similar to AndroSSL where the experiment is fully auto-
mated and no security experts are required. However, the methodology of SMV-Hunter
differs from AndroSSL making the two approaches complementary. SMV-Hunter’s
fully automated support is a great advantage, especially for large scale analysis. How-
ever, when SMV-Hunter fails (complex UI controls, invalid heuristic path), AndroSSL
could take over to provide a more precise control of the user interaction. AndroSSL pro-
vides a more exhaustive set of test scenarios; using a variety of certificates and different
network configurations (while SMV-Hunter tests for a single attack scenario). Hav-
ing multiple scenarios allows AndroSSL to provide precise information regarding the
certificate validation vulnerabilities in an application. For instance, a few applications
tested were successfully attacked in the dynamic URL setup while the attempts failed
in the static URL setup. This leads to the conclusion that those applications validate the
certificate’s hostname, but do not properly validate the signature chain.

5 Conclusion

The main lesson we learned from our experimentation is that mobile application devel-
opers are not aware of the security challenges they face. Some applications, not even
relying on SSL, are just too easy to attack. Tools such as AndroSSL should facilitate
security testing for mobile app developers.

5.1 Summary Results

Table 1. Result Summary

Network Configurations Test Cases Vulnerable Apps Nb

Logcat 3, 4, 34, 35, 36, 37, 54, 72, 73 9
Actual Server
Network 3, 34, 35, 36, 37, 54, 63, 67, 70, 71 10
Self-Signed
Static URL 14, 16, 46H , 53H , 64, 68, 87, 89 8
Valid
Self-Signed
Dynamic URL Signed by Self-Signed 9, 14, 16, 25, 26, 46H , 53H , 64, 68, 87, 89, 90 12
Signed by Real
Total 25

Table 1 provides a summary of the problems found automatically by AndroSSL

when testing 90 popular Android applications. For each test case, the list of vulnera-
ble apps is given (each app being represented by a numeric ID). Apps in bold were
vulnerable to a single experiment (this illustrates the need for a multi test-cases ap-
proach), and subscript H indicates apps hashing the password client side (no salt). The
apps appearing more than once (e.g., 14 and 34) have multiple weaknesses (14 does
not validate the certificate hostname nor the chain while 34 sends the clear password
into the Android log and also on the network). Overall, AndroSSL determined that 25
apps have a vulnerability. If we remove the apps that are just sending the password into
the Android log8 , we are still left with 22. 10 apps sent the user credential through the
network over a non-encrypted channel. Another 12 apps have a broken SSL certificate
validation procedure (8 accepting certificate with an invalid hostname and an invalid
chain while the other 4 accept certificates with an invalid chain as long as they have the
proper hostname).

5.2 Future Work

Several tasks are planned to continue the development of AndroSSL:
– Adding new network configurations is a definitive priority, particularly to have the
possibility to hijack SSL connections only (and target only 1 URL) while letting
non-SSL connections flow towards the legitimate server. This will provide a better
understanding regarding the conditions required for a successful MitM attack. This
should allow for more successful attacks.
– Adding new test cases (i.e., new "fake" certificates) is an ongoing task as there are
many subtle cases worth testing.

References
1. Brubaker, C., Jana, S., Ray, B., Khurshid, S., Shmatikov, V.: Using frankencerts for automated
adversarial testing of certificate validation in SSL/TLS implementations. In: proceedings of
the 2014 IEEE Symposium on Security and Privacy (SP) (2014)
2. Brubaker, C., Klyubin, A., Condra, G.: nogotofail. https://github.com/google/
nogotofail (2014)
3. Dierks, T., Rescorla, E.: Rfc5246 tls v1.2. https://tools.ietf.org/html/rfc5246
(2008)
4. Fahl, S., Harbach, M., Muders, T., Smith, M., Baumgartner, L., Freisleben, B.: Why Eve and
Mallory love Android: An analysis of Android SSL (in)security. In: Proceedings of the 2012
ACM Conference on Computer and Communications Security (CSS’12). pp. 50–61 (2012)
5. Fahl, S., Harbach, M., Perl, H., Koetter, M., Smith, M.: Rethinking SSL development in an
appified world. In: Proceedings of the 2013 ACM Conference on COmputer and Communi-
cations Security (CSS’13). pp. 49–60 (2013)
6. Georgiev, M., Iyengar, S., Jana, S., Anubhai, R., Boneh, D., Shmatikov, V.: The most danger-
ous code in the world: Validating SSL certificate in non-browser software. In: Proceedings of
the 2012 ACM Conference on COmputer and Communications Security (CSS’12). pp. 38–49
(2012)
7. Onwuzurike, L., Cristofaro, E.D.: Danger is my middle name: Experimenting with SSL vul-
nerabilities in Android apps. In: proceedings of the 2015 ACM WiSec (2015)
8. Sillars, D.: Using nogotofail to find issues with your https connections. http:
//developerboards.att.lithium.com/t5/AT-T-Developer-Program-Blogs/Using-
nogotofail-to-Find-Issues-with-Your-HTTPS-Connections/ba-p/39891 (2015)
9. Sounthiraraj, D., Sahs, J., Lin, Z., Khan, L., Greenwood, G.: SMV-Hunter: Large scale, auto-
mated detection of SSL/TLS man-in-the-middle vulnerabilities in Android apps. In: proceed-
ings of the 2014 Network and Distributed System Security Symposium (NDSS’14) (2014)

8 Although this is problematic and should be addressed, we could consider this flaw as more
difficult to exploit.