[Openmcl-devel] Extracting unicode from an external source via FFI
john at mcaleely.com
Sat Feb 21 20:07:07 EST 2009
I'm attempting to get some unicode strings from an external source (a
MySQL database) into a form I can use within CCL (This would be most
convenient if the c data 'became' a native lisp string). I am having
problems with reading them in, and want to ask what the options are
within the CCL FFI. If anyone's been down this route before, I'd be
grateful for pointers.
Welcome to Clozure Common Lisp Version 1.2-r72:73M-ccl (DarwinX8664)!
(Note that the revision number reflects storage in my own subversion
repository. I'm using an unmodified, locally built, version synced
about a month ago.)
My investigations to date (I'm also using clsql 4.0.3/uffi 1.6.0)
suggest that data can make it from a lisp string into the SQL database
(how I've not looked into yet - but the mysql command line sees the
data correctly). When strings come back across the connection, they
arrive garbled. A two character Chinese string in the SQL table
becomes a six character lisp string.
Rummaging into CLSQL/UFFI, I think that ultimately this bit of code
reads strings from the mysql c interface:
#+openmcl ,@(if length
`((ccl:%str-from-ptr ,stored-obj ,length))
Having looked at the ccl code, there is a function near %get-cstring
(defun %get-utf-8-cstring (pointer) ....)
This seems interesting. I speculate:
+ The mysql_c interface is sending over c-style strings, in a
character set of its choice.
+ The uffi code chooses to read this with %get-cstring, which chops
the string into 8 bit bytes, and assumes each is one is a character in
some 256 element character set.
+ The CLSQL code then takes this and passes it back to me as a lisp
+ I wonder if I could convince mysql to use utf-8 within its c
strings, and the uffi code to use %get-utf-8-cstring, then I could
successfully read unicode from the database into lisp strings?
So, if you've been down a similar path, does my speculation sound
Does my gentle use of grep and google appear to have tumbled on the
'right' CCL functions for this work. Is there a 'better' CCL API for
reading a foreign string in some unicode character set?
Looking at the API docs, telling MySQL to use UTF8 seems
straightforward, and I'm willing to hack UFFI/CLSQL to make this work.
Before I start hacking, I thought I'd ask what my options are for
interfacing into CCL's unicode support.
More information about the Openmcl-devel