Bringing Telecommunications In-House - Part 2:
Asterisk 13 LTS

The second part in the series bringing my telecommunications in-house. This article will cover installing the latest Asterisk LTS release.

Asterisk 13 LTS

This article was originally going to be about Kamailio, but my third attempt at getting it to work did not result in anything different happening. I am giving up trying to Kamailio to work for the time being, and instead am going to focus on installing Asterisk 13, the latest LTS release.

Previously I was of the opinion that LTS is better for stability. That turned out the be quite true of Debian, with Debian Jessie completely messing up my install and me having to restore /, /usr, /var, and /etc from backup (i.e. everything except /opt and /home).

Asterisk 11 has some support for SIP SIMPLE messaging, but doesn't regosnise sips: addresses. The Asterisk source code includes a check for sip:/sips:, but I can't determine when that change occurred. Asterisk 12 is a Standard release, and Asterisk 13 is "the next" LTS release. I am, therefore, going to go with Asterisk 13.

A point I should make, however, is that because my restoration did not overwrite /opt, I do still have /opt/certified-asterisk-11.6-cert8/ that I could install from. But I'm not going to, unless Asterisk 13 doesn't work.

Installing Asterisk 13 From Source

sudo mkdir -p /opt/asterisk/asterisk-svn
cd /opt/asterisk/
sudo chown thejc:thejc asterisk-svn
cd asterisk-svn
svn ls http://svn.asterisk.org/svn/asterisk/branches
mkdir asterisk-13-svn
svn checkout http://svn.asterisk.org/svn/asterisk/branches/13 asterisk-13-svn

So we have checked out the source code of asterisk branch 13 into /opt/asterisk/asterisk-svn/asterisk-13-svn/. The reason for the long path is because I might otherwise forget whether it is svn, git, tar, etc.

Now, time to install prerequesites.

cd asterisk-13-svn
./configure
sudo apt-get install uuid-dev libsqlite3-dev
./configure
sudo apt-get install libjansson-dev libxslt-dev
./configure

configure: Menuselect build configuration successfully completed

               .$$$$$$$$$$$$$$$=..
            .$7$7..          .7$$7:.
          .$$:.                 ,$7.7
        .$7.     7$$$$           .$$77
     ..$$.       $$$$$            .$$$7
    ..7$   .?.   $$$$$   .?.       7$$$.
   $.$.   .$$$7. $$$$7 .7$$$.      .$$$.
 .777.   .$$$$$$77$$$77$$$$$7.      $$$,
 $$$~      .7$$$$$$$$$$$$$7.       .$$$.
.$$7          .7$$$$$$$7:          ?$$$.
$$$          ?7$$$$$$$$$$I        .$$$7
$$$       .7$$$$$$$$$$$$$$$$      :$$$.
$$$       $$$$$$7$$$$$$$$$$$$    .$$$.
$$$        $$$   7$$$7  .$$$    .$$$.
$$$$             $$$$7         .$$$.
7$$$7            7$$$$        7$$$
 $$$$$                        $$$
  $$$$7.                       $$  (TM)
   $$$$$$$.           .7$$$$$$  $$
     $$$$$$$$$$$$7$$$$$$$$$.$$$$$$
       $$$$$$$$$$$$$$$$.

configure: Package configured for:
configure: OS type  : linux-gnu
configure: Host CPU : x86_64
configure: build-cpu:vendor:os: x86_64 : unknown : linux-gnu :
configure: host-cpu:vendor:os: x86_64 : unknown : linux-gnu :

make menuselect

As this is not a certified version of Asterisk, everything with satisfied dependencies will be installed. I have added the following, however:

Core Sound Packages
- ~~CORE-SOUNDS-EN-WAV~~ CORE-SOUNDS-EN-GSM
Extra Sound Packages
- EXTRA-SOUNDS-EN-GSM

One thing I noted while looking through menuselect: pjsip stuff isn't going to be installed. I am sure I read somewhere that a lot of stuff now relies on it. I might switch from chap_sip to chan_pjsip, but I am currently undecided. Anyway, the PJSIP Channel Driver won't be installed because it depends on pjproject (E), res_pjsip (M), and res_pjsip_session (M). I need to find pjproject, and apt-cache search suggests the one it is most likely to be is libpjproject-dev.

sudo apt-get install libpjproject-dev
sudo apt-get remove asterisk
./configure
make menuconfig

I think that looks OK.

make -j 2
sudo apt-get install libsrtp0-dev
make -j 2
sudo make install -j 2
sudo apt-get remove asterisk-modules
sudo make install -j 2
sudo make samples
sudo apt-get install doxygen
./configure
make -j2
sudo make progdocs

Asterisk 13 LTS is now installed, however it is not configured or running yet.

New(er) Configuration Files

I have made some changes to my sip.conf and extensions.conf since the last article, that I restored from my backup. I am, however, going to check I can just restore them first.

TLS Setup

In Asterisk 1.8 and 11, TLS configuration was pretty much identical.

sudo nano /etc/asterisk/sip.conf

match_auth_username=yes
realm=sip.thejc.me.uk
tlsenable=yes
srvlookup=no
timezone=uk
dtmfmode = rfc2833
compactheaders = yes
alwaysauthreject = yes
accept_outofcall_messages = yes
outofcall_message_context = messages
auth_message_requests = yes
tlscertfile=/etc/ssl/asterisk/sip_thejc_me_uk.pem
tlsprivatekey=/etc/ssl/asterisk/sip_thejc_me_uk.key
tlscapath=/etc/ssl/certs
tlsdontverifyserver=yes
tlscipher=ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
tlsclientmethod=tlsv1
domain=sip.thejc.me.uk
fromdomain=sip.thejc.me.uk

[basic-options](!)
context=internal

[my-codecs](!)
disallow=all
allow=ulaw
allow=alaw

[groundwire-codecs](!)
disallow=all
allow=g722
allow=ulaw
allow=alaw
allow=g729
allow=gsm
allow=ilbc

[tls-phone](!)
transport = tls

[6001](natted-phone,groundwire-codecs,tls-phone)
md5secret = [md5hashOf6001:pw:sip.thejc.me.uk]

[6002](natted-phone,groundwire-codecs,tls-phone)
md5secret = [md5hashOf6002:pw:sip.thejc.me.uk]

sudo nano /etc/asterisk/rtp.conf

rtpstart=15000
rtpend=19999

sudo asterisk -rvvv
module reload

At this point, check 6001 and 6002 can register, which they can.

sudo nano /etc/asterisk/extensions.conf

sudo su
mv /etc/asterisk/extensions.conf /etc/asterisk/extensions.conf.orig.v11
cp /home/thejc/extensions.conf.bak /etc/asterisk/extensions.conf
exit
sudo asterisk -rvvv
dialplan reload

Try dialling extension 6002 from extension 6001: successful, two way audio.

Try dialling extension 6001 from extension 6002: successful, but one way audio.

By removing g729 from [groundwire-codecs], all calls are successful with two-way audio.

The next number to try dialling is 0800500005, which is the BT Automated News phone number. If my dialplan (and Groundwire settings) are correct, the number should get converted to +44800500005 and go out through Localphone. Well, it would if it could dial out.

sudo nano /etc/asterisk/sip.conf

[localphone]
...

[sipgate]
...

[netsip]
...

[6001](natted-phone,groundwire-codecs,tls-phone)
md5secret = [md5hashOf6001:pw:sip.thejc.me.uk]
context = external-permitted

[6002](natted-phone,groundwire-codecs,tls-phone)
md5secret = [md5hashOf6002:pw:sip.thejc.me.uk]
context = external-permitted

At this point, everything seems to be working properly. The only issue is a load of WARNING[...]: db.c:332 ast_db_put: Couldn't execute statement: SQL logic error or missing database warnings/errors, but that is probably because I'm not using it.

More Testing of Calls and Messages

It appears that sips is still not recognised, as I'm getting WARNING[...][...]: message.c:1223 msg_send_exec: No message technology 'sips' found. Thus, I cannot yet get rid of my lines of code transforming sips: to sip:

sudo nano /etc/asterisk/extensions.conf

[messages]
exten => _X.,1,Verbose(1,SMS Receiving Dial Plan invoked...)
same => n,Verbose(1,To: ${MESSAGE(to)})
same => n,Verbose(1,From: ${MESSAGE(from)})
same => n,Verbose(1,Body: ${MESSAGE(body)})
same => n,Set(ACTUALTO=${CUT(MESSAGE(to),@,1)})
same => n,GotoIf($["${ACTUALTO:0:5}" == "sips:"]?7:8)
same => n,Set(ACTUALTO=sip:${ACTUALTO:5})
same => n,MessageSend(${ACTUALTO},${MESSAGE(from)})
same => n,Verbose(1,Send status is: ${MESSAGE_SEND_STATUS})
same => n,GotoIf($["${MESSAGE_SEND_STATUS}" != "SUCCESS"]?sendfailedmsg)
same => n,Hangup()
; Handle failed message
same => n(sendfailedmsg),Set(MESSAGE(body)="[${STRFTIME(${EPOCH},,%d%m%Y-%H:%M:%S)}] Your  message to ${EXTEN} has failed. Retry later.")
same => n,Set(ME_1=${CUT(MESSAGE(from),<,2)})
same => n,Set(ACTUALFROM=${CUT(ME_1,@,1)})
same => n,GotoIf($["${ACTUALFROM:0:5}" == "sips:"]?16:17)
same => n,Set(ACTUALFROM=sip:${ACTUALFROM:5})
same => n,MessageSend(${ACTUALFROM},ServiceCenter)
same => n,Hangup()
same => n,Hangup()

This code basically uses SIP SIMPLE to send internal messages between extensions. The first GotoIf checks if the destination address begins sips:, and if it does the next line transforms it to sip:.

The one problem with Groundwire is that of a contact having more than one sip:/sips:/groundwire:/asoftphone: URI and/or mobile number associated with it, as it merges all the contacts together. Because of that, I am not (yet) setting up Asterisk to send messages externally.

My current workaround is that Groundwire has two profiles - one with SIP SIMPLE enabled, and one with SIP SIMPLE disabled but SMS enabled.

If my SMS API script receives a POST request and the destination number begins with sip, then it returns a 400 error. If my Asterisk SIP SIMPLE processing receives a message that is not addressed to a valid extension (valid extensions begin with a 6, not a +) then Groundwire returns a "Messaging error".

At the bottom right of Groundwire, just above the Send button, is "John Cook" (for SIP SIMPLE messaging) which if tapped brings up the option for "John Cook", "SMS", or "Localphone" (with Localphone being my previous test of SIP SIMPLE messaging). If I experience an error sending a message, because I'm using the wrong profile, it is a simple two taps to change method of sending.

Something also worth mentioning is that if it weren't for my home server going down during that upgrade, I wouldn't have realised that my SMS API didn't work externally over 3G. One of the things I did as soon as I restored my Web server configuration was to open port 443 in my home server firewall.

As I am using a PAYG SIM card in my E220 modem, I don't have a massive worry about a potential hack causing huge financial loss. It might cause inconvenience because I wouldn't be able to send text messages, but the most I could lose is the credit on the SIM.

As for the Web server setup, I had to use lighttpd as well as nginx because nginx alone is pretty useless at PHP (or is just too difficult for me to setup). So, here is my /etc/nginx/nginx.conf file (where I have modified it):

http {

upstream sip.lighttpd {
        server 192.168.1.20:8090;
}

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

map $spdy $spdy_ae {
        default  $http_accept_encoding;
#       default "gzip";
        2        "gzip, deflate";
        3        "gzip, deflate";
        3.1        "gzip, deflate";
    }

map $spdy $spdy_connection {
        default "0";
        2       "1";
        3       "1";
        3.1     "1";
}

Basically, those are inside the http { block. And here is my /etc/nginx/conf.d/example_ssl.conf:

server {
        listen 443 ssl spdy;
        server_name sip.thejc.me.uk;
        ssl_certificate         /etc/ssl/asterisk/sip_thejc_me_uk.pem;
        ssl_certificate_key     /etc/ssl/asterisk/sip_thejc_me_uk.key;

        ssl_session_cache       shared:SSL:1m;
        ssl_session_timeout     5m;

        ssl_ciphers             "ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS";

        location / {
                proxy_pass      http://sip.lighttpd;
                proxy_set_header        Accept-Encoding $spdy_ae;
                proxy_set_header        Host    $host;
                proxy_set_header        X-Real-IP       $remote_addr;
                proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_pass_request_headers      on;
        }
}

As for my lighttpd config, I am using my standard setup and have just added the following to one of my existing vhosts.d/ files:

$SERVER["socket"] == "192.168.1.20:8090" {
        var.servername = "/sip.thejc.me.uk"
        server.document-root = var.basedir + servername
}

Then, I just have /sip.thejc.me.uk/sms.php containing the following in my Web server (redacted where appropriate):

<?php
include 'sms_inject.php';
// username = %account[username]%
// password = %account[password]%
// message = %sms_body%
// to = %sms_to%
// from = %account[userCallerId]%

// u = %account[username]%
// p = %account[password]%
// sms = %sms_body%
// to = %sms_to%
// from = %account[userCallerId]%

// https://sip.thejc.me.uk/sms.php?u=blah&p=blah&from=blah&to=blah&sms=blahblahblah
if (isset($_POST["u"]) && isset($_POST["p"]) && isset($_POST["msg"]) && isset($_POST["to"])) {
        $u = $_POST["u"];
        $p = $_POST["p"];
        $sms = $_POST["msg"];
        $sms = str_replace(chr(194).chr(163),chr(163),$sms);
        $sms = str_replace(chr(194).chr(165),chr(165),$sms);
        $sms = str_replace(chr(226).chr(130).chr(172),"EUR",$sms);
        $sms = str_replace(chr(226).chr(128).chr(162),"*",$sms);
        $sms = mysql_escape_string($sms);
        $to = rawurldecode($_POST["to"]);

        if (($u == "redacted-username") && ($p == "redacted-password") && ($sms != "%sms_body%") && ($to != "%sms_to%") && (substr($to,0,3) != "sip")) {
                $mysql_resource = mysql_connect('127.0.0.1','smsd','redacted-mysql-password');
                mysql_select_db('smsd',$mysql_resource);
                $smsd=new sms_inject($mysql_resource);
                $smsd->send_sms($sms,$to);
                echo "<response><error>0</error><description>Success</description></response>";
        } else {
                http_response_code(400);
                echo "<response><error>400</error><description>Invalid parameters</description></response>";
        }
} else {
http_response_code(400);
echo "<response><error>400</error><description>Invalid parameters</description></response>"
;
}
?>

And then in Groundwire SMS Profile/Account, I have the URL https://sip.thejc.me.uk/sms.php and in the POST box I have to=%sms_to%&msg=%sms_body%&u=redacted-username&p=redacted-password - it took some trial and error (the reason for the checks of value of $sms and $to) until I discovered that the substitutions only work if the %string_to_substitute% variables came before fixed strings (like u= and p=).

As for the included file, sms_inject.php, that is "gammu smsd class" by Ikhsan Agustian with a GNU/GPL licence, available from Gammu SMS Inject: Send SMS messages using Gammu MySQL storage.

The final things that are needed are sudo apt-get install gammu-smsd, and sudo nano /etc/gammu-smsdrc.

[gammu]
port = /dev/ttyUSB0
connection = at115200

[smsd]
service = SQL
driver = native_mysql
host = 127.0.0.1
user = mysql_username_for_smsd
password = mysql_password_for_smsd
database = smsd
logfile = syslog
debuglevel = 0

inboxpath = /var/spool/gammu/inbox/
outboxpath = /var/spool/gammu/outbox/
sentsmspath = /var/spool/gammu/sent/
errorsmspath = /var/spool/gammu/error/

The final thing needed is setting up the database username, password, database, tables, and permissions. I won't cover that here, though.

Although this could be setup for multiple users, multiple modems and SIM cards, multiple SMS CLIDs, et cetera, I am only using SMS for myself.

The one issue remaining, though, is that although incoming SMS messages get stored in the smsd database, I have no way (other than using mysql commands) of reading the messages, and getting incoming messages to my phone is not currently possible.

gammu-smsd does have the option of running a script on each incoming message, but the current question is "how will I get that into Asterisk?"

What I ideally need is a CLI program that can send SIP SIMPLE messages, or an XMPP CLI client that can send encrypted XMPP messages (i.e. the choice between them showing up on my phone in either Groundwire or ChatSecure). ChatSecure has a timeout, though, so I need SIP SIMPLE. This is where I hit my previous problem: I can't get any SIP SIMPLE compatible CLI clients to install/work on Debian Wheezy.

Something that should be pointed out is that I had to create a hack so that my SMS script works as good as it could, temporarily. Something is broken somewhere with the code pages, either in PHP processing the POST data, the fact SMS uses a different code page, or something else, but the temporary fix I previously used of converting from "8bit" to "UTF-8" meant that the only characters that then got corrupted (excluding special characters that require holding a key) are two on a UK iPhone keyboard - the Euro symbol (which becomes ¬) and the dot symbol to the right of the Yen symbol which becomes a double quote.

After further examination, I can convert the £ and ¥ symbols using str_replace, but the € and • symbols are not so easy. As € takes up 3 bytes in UTF-8, as does •, I have replaced € with EUR and • with *. It is not ideal, but it now means virtually all texts I could send are at least readable and make sense. I'll investigate it at some point in the future, but at least for the time being it isn't that much of an issue for me.

I am wondering if I can somehow create a .call file for Asterisk upon receiving an SMS, and it does appear that I can.

Incoming SMS Becomes SIP SIMPLE Message

Some of the credit for this has to go to Asterisk 10 or 11 Messaging (SMS/SIP Messaging) with offline message sending, because it gave me the template for how to do what I have spent several days trying to achieve.

First, .call files need to "call" someone, so in /etc/asterisk/extensions.conf I need to create a "fake" call answering extension.

[app-fakeanswer]
exten => 6000,1,Answer()
exten => 6000,n,Wait(1)
exten => 6000,n,Hangup()

Then, I need to create a file in /var/spool/asterisk/tmp/ with a .call extension.

cd /var/spool/asterisk/tmp
sudo mkdir /var/spool/asterisk/outgoing_done
sudo chown asterisk:asterisk /var/spool/asterisk/outgoing_done
sudo nano "`date --rfc-3339=ns`.call"

Channel: Local/6000@app-fakeanswer
Priority: 1
Context: messages
Set: MESSAGE(to)=sip:6001@sip.thejc.me.uk
Set: MESSAGE(from)=+447[mobile number]
Set: MESSAGE(body)=This is a test message.
Extension: 6001
Archive: Yes

sudo chown asterisk:asterisk [tab]
sudo mv [tab] ../outgoing/

Because MESSAGE(from) is set to a mobile number, push notifications do display the number rather than a contact name if one exists. However, if a contact exists, as soon as Groundwire is opened the message appears from the named contact rather than the number that was displayed in the push notification. Also, the time received is the time the message (rather than push notification) was received, so it is more of a "time read" that "time sent" indicator.

Now, this isn't pretty because of how bad the parsing of call files in Asterisk is. I have managed to escape semicolons and forward slashes so they appear properly, but hash symbols retain a slash when sent to the phone.

nano /home/thejc/Scripts/sms-receive-1.sh

#!/bin/sh
PROGRAM=/home/thejc/Scripts/sms-receive-2.sh
for i in `seq $SMS_MESSAGES` ; do
    eval "$PROGRAM \"\${SMS_${i}_NUMBER}\" \"\${SMS_${i}_TEXT}\""
done

sudo usermod -a -G thejc asterisk
sudo usermod -a -G gammu asterisk
nano /home/thejc/Scripts/sms-receive-2.sh

#!/bin/sh
filename="`date --rfc-3339=ns`.call"
tempdir="/var/spool/asterisk/tmp/"
outdir="/var/spool/asterisk/outgoing/"
smssender="$1"
smstext="$2"
cat << EOF > "$tempdir$filename"
Channel: Local/6000@app-fakeanswer
Priority: 1
Context: messages
Set: MESSAGE(to)=sip:6001@sip.thejc.me.uk
Set: MESSAGE(from)="$smssender"
Set: MESSAGE(body)="$smstext"
Extension: 6001
Archive: Yes
EOF
sed -i 's/[\]/\\\\/' "$tempdir$filename"
sed -i 's/[;]/\\\;/' "$tempdir$filename"
sed -i 's/[#]/\\\#/' "$tempdir$filename"
mv "$tempdir$filename" "$outdir$filename"

sudo chown asterisk:asterisk /var/spool/asterisk/outgoing/ /var/spool/asterisk/outgoing_done/ /var/spool/asterisk/tmp/
sudo service gammu-smsd restart

At this point, SMS messages received by the SIM in my E220 are relayed over SIP SIMPLE to Groundwire, and replies using the SMS profile/account are sent via HTTPS POST.

This is far from perfect, and it doesn't support multi-part SMS messages being sent as a single message (couldn't get the sample code to work), and there are bugs with escaping characters and character encoding issues too.

But, I can now receive text messages on my mobile without having to pay AAISP for receiving them (other than for the data usage). Speaking of, I have received an e-mail stating my SIP2SIM has been sent first class, so will hopefully be with me tomorrow.

Receive Calls Through GSM Modem?

I wonder if it is possible to receive phone calls using an E220 modem. According to Caller ID with Linux and Huawei e220 it might be possible to wait for an incoming call and print the caller ID of the number.

That then begs the question: can I proxy an incoming call over Asterisk? As a Nominet registrar, I could contact Nominet and find out what the requirements are for becoming an E.164 +44 ENUM registrar, which would allow some calls to go over SIP, but could I get calls to come in over GSM and then patch them through Asterisk? AT^CVOICE? suggests it is not possible, at least with the current firmware in my dongle.

Oh well. I don't really want to spend more money just to enable voice calls, especially when they are 3p/min which is more expensive than calling landlines using netSIP/Localphone and although 1.2p/min cheaper than calling mobiles using SIP2SIM+Localphone, I would have to make 83 minutes 20 seconds of calls to mobiles from my SIP2SIM mobile just to cover 1.00 GBP of the cost of a newer dongle. Even if a suitable unlocked (any mobile network) modem cost only 12.00 GBP, it would likely take more than 2 years to break even.

For the time being, using my mobile number just for SMS should be OK. Most companies that ask for a mobile number ask for a "landline" as well, and at the moment my new mobile number (not transferred old one over from O2 yet) just rings as I have voicemail disabled.

UK Ring Tone

One last thing I want to cover in this article, switching from a US ringing tone to a UK rining tone.

sudo nano /etc/asterisk/indications.conf

[general]
country=uk

sudo asterisk -r

module reload

Although this doesn't work for Groundwire, it will hopefully work when my D70 arrives (although I expect I will have to configure it anyway).

That is the end of this article. The next one in the series will probably cover setting up SIP2SIM, or setting up my D70, whichever comes first.