Bug #1591
closedlibdbi is buggy and slow, get rid of it
Added by laforge about 8 years ago. Updated about 6 years ago.
80%
Description
libdbi seemed like a good idea at the time, but in reality we always only used the sqlite3 backend, and libdbi causes more trouble than advantages. Both in terms of bugs, as well as performance. We should simply use sqlite3 directly.
And of coure move to an asynchronous database interface, too - but that's a separate topic.
Related issues
Updated by laforge about 8 years ago
migrated to https://projects.osmocom.org/issues/1591
Updated by laforge about 8 years ago
- Related to Feature #1592: VLR in libmsc, to connect to HLR asynchronously added
Updated by laforge over 7 years ago
- Target version set to Asynchronous HLR+AUC for CS
Updated by neels over 7 years ago
I take it that once libvlr is in place, there will be no libdbi in osmo-nitb/osmo-cscn.
-> watch #1592
Updated by neels over 7 years ago
I'm noticing that we apparently do still need the db for SMS.
So what was previously the hlr.sqlite3 database is mostly replaced by libvlr,
but it will remain as an sms.sqlite3.
Am I missing something?
Updated by laforge over 7 years ago
On Wed, Jan 04, 2017 at 04:11:00PM +0000, neels [REDMINE] wrote:
I'm noticing that we apparently do still need the db for SMS.
So what was previously the hlr.sqlite3 database is mostly replaced by libvlr,
but it will remain as an sms.sqlite3.
yes, until the SMSC has been externalized [if needed] at some later
point. It could e.g. be comparable to MNCC: A run-time option whether
to run with the internal handler, or to use SMPP in transaction mode
only without any internal store-and-forward?
From past experience: The SMS load in practise has not been that much of
an issue compared to the "HLR" load.
--
- Harald Welte <laforge@gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
Updated by neels over 7 years ago
- Related to Bug #30: sqlite3 database / asynchronous access to it added
Updated by neels over 7 years ago
- Blocks Support #1923: re-enable all nightly package builds added
Updated by stuge over 7 years ago
I started looking into this several years ago and I'd still like to help if I can.
Maybe I can't help code within your required timeframe, but I would love to at least have a brainstorming/design discussion - ideally in person, but online works too.
I can contribute experience with SQLite, MySQL/MariaDB and PostgreSQL in C and SQL-fu.
Updated by msuraev over 7 years ago
- Related to Feature #1860: Include Ubuntu 16.04 LTS in nightly builds added
Updated by neels over 7 years ago
Hi stuge!
Help is always welcome.
This change must be seen in the light of #1592, IOW the neels/vlr branch (based on laforge's VLR work) that pretty much replaces half of our MSC with a brand new VLR implementation and an external OsmoHLR process.
The point is that the sqlite file that OsmoNITB keeps will not store anything about subscribers anymore. It will only be used for SMS, if at all.
Let me paste a relevant snippet from #1592:
With SMS, the problem is still the legacy db. Obtaining an SMS from the DB entails a JOIN with the no longer
populated Subscriber table (to pick only those that are active, i.e. lac >= 0).
I'll have to rethink the SMS code.How to deal with SMS that have no active subscriber: in the long term we will need a plan
for undelivered SMS [whose recipients] show up with a different VLR, so more of a quick hack is
probably sufficient [for local delivery] at this point, as long as we assume that there is only one VLR.The SMS code should also be moved away from libdbi to native sqlite.
I will see whether doing both changes in one go makes sense.
The database file should probably also be renamed to sms.db.
Questions to clarify:
- what to do with SMS in general? I'm not at all familiar with SMS delivery beyond the simple case where sender and recipient are both actively available at the same OsmoNITB. What's the roadmap? Do we even need SMS storage in a DB, or can we live with an in-memory
llist
storage until the bigger picture resolves? - Given that we still need to keep SMS in a DB in OsmoNITB, we need to adjust the code that looks for pending SMS. So far it could SELECT only those SMS with a recipient that is actively attached (by querying "lac > 0"). Now we'd have to SELECT one or more SMS and in a second step look up whether their recipients are currently active in the VLR. Not hard, but harder to make efficient.
- Finally, given that we keep SMS in a DB, and while adjusting the SQL anyway, we could also move away from libdbi in one fell swoop. <-- and this point is what this issue is about.
If you have any ideas or patches for the openbsc.git:neels/vlr branch, they would be very welcome.
Updated by laforge over 7 years ago
On Fri, Jan 20, 2017 at 10:56:48PM +0000, neels [REDMINE] wrote:
- what to do with SMS in general? I'm not at all familiar with SMS
delivery beyond the simple case where sender and recipient are both
actively available at the same OsmoNITB. What's the roadmap?
See below.
Do we even need SMS storage in a DB, or can we live with an
in-memoryllist
storage until the bigger picture resolves?
no, the latter is clearly not sufficient. A persistent database is
needed.
However, that database should sooner or later not be part of the NITB as
the NITB is moving towards becoming a classic VLR+MSC only. At that
point, all SMS should be handled by an external process.
In general I would like to see SMPP to be used here, but unfortunately
there is somewhat of an "impedacne mis-match" between what is required
and what SMPP is designed for: SMPP is intended for communication of an
SMSC with external ESMEs (sms receiving/sending applications), and not
for the communication between the MSC and the SMSC.
The Communication between MSC and SMSC (which is what we're looking at
here) is normally done over MAP, using procedures like MO-forward-SM to
the GT of the SMSC. It's thus on the same level as GSUP.
- do we want to use SMPP for the MSC-SMSC interface despite its
inadequacies, maybe with some custom extensions? - do we want to introduce a new protocol?
- do we want to hack up GSUP to include this? (probably not)
Please also note that Fairwaves was working on something in this area
which they then used to convert SMS to SIP, AFAIR.
- Given that we still need to keep SMS in a DB in OsmoNITB, we need to adjust the code that looks for pending SMS. So far it could SELECT only those SMS with a recipient that is actively attached (by querying "lac > 0"). Now we'd have to SELECT one or more SMS and in a second step look up whether their recipients are currently active in the VLR. Not hard, but harder to make efficient.
I wouldn't worry about efficiency at this point.
The short term goal is to get the HLR/VLR split done and spend as little
as possible on side-track issues like SMS.
- Finally, given that we keep SMS in a DB, and while adjusting the SQL anyway, we could also move away from libdbi in one fell swoop. <-- and this point is what this issue is about.
I am wondering if that intermediate step is really useful. The question
is how long the sms handling will remain within the NITB, before being
externalized.
Updated by neels about 7 years ago
I think I will implement lookup code that finds pending SMS for active subscribers
by looking in the VLR for active subscribers (RAM) and matching that up with pending
SMS recipients (SQL), to get basic functionality disregarding efficiency. (later)
Updated by neels about 7 years ago
(on neels/vlr branch, SMS lookup is now done by round-robin on receiver MSISDN by SQL in the SMS db
and per hit a lookup in the VLR subscribers list in RAM)
Updated by keith over 6 years ago
laforge wrote:
Please also note that Fairwaves was working on something in this area
which they then used to convert SMS to SIP, AFAIR.
Could anybody from fairwaves point to branch/commits related to this?
Updated by ipse over 6 years ago
keith wrote:
laforge wrote:
Please also note that Fairwaves was working on something in this area
which they then used to convert SMS to SIP, AFAIR.Could anybody from fairwaves point to branch/commits related to this?
See a discussion on the mailing list about external USSD. We have only one active branch (per repo) which has all our changes in it. So it's the same branch with both external USSD, SMS, and LURs.
I think we should create a separate ticket for the external SMS functionality like this one for external USSD support: https://osmocom.org/issues/1597
Updated by laforge over 6 years ago
- Target version deleted (
Asynchronous HLR+AUC for CS)
Updated by laforge over 6 years ago
- Blocks deleted (Support #1923: re-enable all nightly package builds)
Updated by laforge over 6 years ago
- Related to Support #1923: re-enable all nightly package builds added
IMPORTANT NOTICE: This page contains information about a legacy version of the Osmocom software. This legacy version is no longer maintained. If you use it, don't be surprised if it doesn't work. It was your choice to ignore man-years worth of developments, improvements and fixes. Please migrate to the active/supported software (Osmocom CNI, consisting of OsmoBSC, OsmoMSC, OsmoHLR, OsmoSTP, OsmoMGW - a NITB style setup is described at Osmocom_Network_In_The_Box).