UTF8 Character Encoding

[Daniel Woods]

Hi Everyone,

I realize that this issue has been brought up several times, but I
definitely need some help... I have tried everything that is
recommended in all the docs and lists and previous emails from this
list on the subject.

I have a mysql database that has been created with the charset=utf8. I
have ensured that /etc/my.cnf is properly configured with utf8 and
utf8_general_ci values on the "default-character-set" and
"default-collation" directives respectively. My MySQL JDBC URL string
contains the "useUnicode=true" and "characterEncoding=utf8"
parameters. My Config.groovy contains the "grails.views.gsp.encoding"
and "grails.converters.encoding" parameters with both set to "UTF-8".
I have confirmed that the data in the database is UTF-8 encoded data
and is thereby not dirty. I've done this by 1) performing a simple
command-line dump of a column containing UTF-8 encoded characters into
a file; and 2) attached the old Ruby on Rails app to the database (I
am converting the app from RoR to Grails), and it renders the UTF-8
characters to the browser just fine. I can definitely confirm that
Grails rendering the UTF-8 characters with Latin1 (ISO-8859-1)
encoding. The characters: æ are rendered instead of æ ... I have
tried setting the contentType and characterEncoding parameters on the
response object, both in the view and in the controller... Nothing
works. I've even taken a more Java-esque approach and tried to create
a properly encoded String object using the data I'm getting back from
the database. It is almost as though the MySQL JDBC library is not
properly respecting the useUnicode and characterEncoding parameters.

I don't know what else to try... What can I do here?

Thanks in advance.

Dan

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Enrique Medina Montenegro]

Are you starting up Tomcat within Grails, or deploying the Grails application to a separated Tomcat?

If the latter, try setting "URIEncoding=UTF-8" attribute in Connector tag.

On Tue, Jul 12, 2011 at 3:50 AM, Daniel Woods <[hidden email]> wrote:

Hi Everyone,

I realize that this issue has been brought up several times, but I
definitely need some help... I have tried everything that is
recommended in all the docs and lists and previous emails from this
list on the subject.

I have a mysql database that has been created with the charset=utf8. I
have ensured that /etc/my.cnf is properly configured with utf8 and
utf8_general_ci values on the "default-character-set" and
"default-collation" directives respectively. My MySQL JDBC URL string
contains the "useUnicode=true" and "characterEncoding=utf8"
parameters. My Config.groovy contains the "grails.views.gsp.encoding"
and "grails.converters.encoding" parameters with both set to "UTF-8".
I have confirmed that the data in the database is UTF-8 encoded data
and is thereby not dirty. I've done this by 1) performing a simple
command-line dump of a column containing UTF-8 encoded characters into
a file; and 2) attached the old Ruby on Rails app to the database (I
am converting the app from RoR to Grails), and it renders the UTF-8
characters to the browser just fine. I can definitely confirm that
Grails rendering the UTF-8 characters with Latin1 (ISO-8859-1)
encoding. The characters: æ are rendered instead of æ ... I have
tried setting the contentType and characterEncoding parameters on the
response object, both in the view and in the controller... Nothing
works. I've even taken a more Java-esque approach and tried to create
a properly encoded String object using the data I'm getting back from
the database. It is almost as though the MySQL JDBC library is not
properly respecting the useUnicode and characterEncoding parameters.

I don't know what else to try... What can I do here?

Thanks in advance.

Dan

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email

[pledbrook]

> "default-collation" directives respectively. My MySQL JDBC URL string
> contains the "useUnicode=true" and "characterEncoding=utf8"
> parameters. My Config.groovy contains the "grails.views.gsp.encoding"
> and "grails.converters.encoding" parameters with both set to "UTF-8".

This problem is typically a result of a database or db connection
issue. For what it's worth, the JDBC URL on grails.org is:

    jdbc:mysql://...?autoReconnect=true&useUnicode=yes&characterEncoding=UTF-8

and that seems to be working fine.

Peter

--
Peter Ledbrook
Grails Advocate
SpringSource - A Division of VMware

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Daniel Woods]

> Are you starting up Tomcat within Grails, or deploying the Grails
> application to a separated Tomcat?
> If the latter, try setting "URIEncoding=UTF-8" attribute in Connector tag.

I've tried it with running Tomcat within Grails and as deployed to
Tomcat... Neither makes a difference. I'm actually using Jetty right
now, but this hasn't had an effect one way or another.

> On Tue, Jul 12, 2011 at 3:50 AM, Daniel Woods <[hidden email]>
> wrote:
>>
>> Hi Everyone,
>>
>> I realize that this issue has been brought up several times, but I
>> definitely need some help... I have tried everything that is
>> recommended in all the docs and lists and previous emails from this
>> list on the subject.
>>
>> I have a mysql database that has been created with the charset=utf8. I
>> have ensured that /etc/my.cnf is properly configured with utf8 and
>> utf8_general_ci values on the "default-character-set" and
>> "default-collation" directives respectively. My MySQL JDBC URL string
>> contains the "useUnicode=true" and "characterEncoding=utf8"
>> parameters. My Config.groovy contains the "grails.views.gsp.encoding"
>> and "grails.converters.encoding" parameters with both set to "UTF-8".
>> I have confirmed that the data in the database is UTF-8 encoded data
>> and is thereby not dirty. I've done this by 1) performing a simple
>> command-line dump of a column containing UTF-8 encoded characters into
>> a file; and 2) attached the old Ruby on Rails app to the database (I
>> am converting the app from RoR to Grails), and it renders the UTF-8
>> characters to the browser just fine. I can definitely confirm that
>> Grails rendering the UTF-8 characters with Latin1 (ISO-8859-1)
>> encoding. The characters: æ are rendered instead of æ ... I have
>> tried setting the contentType and characterEncoding parameters on the
>> response object, both in the view and in the controller... Nothing
>> works. I've even taken a more Java-esque approach and tried to create
>> a properly encoded String object using the data I'm getting back from
>> the database. It is almost as though the MySQL JDBC library is not
>> properly respecting the useUnicode and characterEncoding parameters.
>>
>> I don't know what else to try... What can I do here?
>>
>> Thanks in advance.
>>
>> Dan
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>    http://xircles.codehaus.org/manage_email
>>
>>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

I think the problem is between the dump and the import of the data (or the data per si, there's something called signature that i don't know what it is), have you tried with a fresh app (no data import) to see if it can generate the proper characters? After you dumped the file have you checked it? eg: file --mime myfile.sql

On Jul 12, 2011, at 11:21 PM, Daniel Woods wrote:

>> Are you starting up Tomcat within Grails, or deploying the Grails
>> application to a separated Tomcat?
>> If the latter, try setting "URIEncoding=UTF-8" attribute in Connector tag.
>
> I've tried it with running Tomcat within Grails and as deployed to
> Tomcat... Neither makes a difference. I'm actually using Jetty right
> now, but this hasn't had an effect one way or another.
>
>> On Tue, Jul 12, 2011 at 3:50 AM, Daniel Woods <[hidden email]>
>> wrote:
>>>
>>> Hi Everyone,
>>>
>>> I realize that this issue has been brought up several times, but I
>>> definitely need some help... I have tried everything that is
>>> recommended in all the docs and lists and previous emails from this
>>> list on the subject.
>>>
>>> I have a mysql database that has been created with the charset=utf8. I
>>> have ensured that /etc/my.cnf is properly configured with utf8 and
>>> utf8_general_ci values on the "default-character-set" and
>>> "default-collation" directives respectively. My MySQL JDBC URL string
>>> contains the "useUnicode=true" and "characterEncoding=utf8"
>>> parameters. My Config.groovy contains the "grails.views.gsp.encoding"
>>> and "grails.converters.encoding" parameters with both set to "UTF-8".
>>> I have confirmed that the data in the database is UTF-8 encoded data
>>> and is thereby not dirty. I've done this by 1) performing a simple
>>> command-line dump of a column containing UTF-8 encoded characters into
>>> a file; and 2) attached the old Ruby on Rails app to the database (I
>>> am converting the app from RoR to Grails), and it renders the UTF-8
>>> characters to the browser just fine. I can definitely confirm that
>>> Grails rendering the UTF-8 characters with Latin1 (ISO-8859-1)
>>> encoding. The characters: æ are rendered instead of æ ... I have
>>> tried setting the contentType and characterEncoding parameters on the
>>> response object, both in the view and in the controller... Nothing
>>> works. I've even taken a more Java-esque approach and tried to create
>>> a properly encoded String object using the data I'm getting back from
>>> the database. It is almost as though the MySQL JDBC library is not
>>> properly respecting the useUnicode and characterEncoding parameters.
>>>
>>> I don't know what else to try... What can I do here?
>>>
>>> Thanks in advance.
>>>
>>> Dan
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe from this list, please visit:
>>>
>>>    http://xircles.codehaus.org/manage_email
>>>
>>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Daniel Woods]

> I think the problem is between the dump and the import of the data (or the data per si, there's something called signature that i don't know what it is), have you tried with a fresh app (no data import) to see if it can generate the proper characters? After you dumped the file have you checked it? eg: file --mime myfile.sql
>

I'm not sure what "signature" is either as it pertains to character
encoding. I've not done any data import; I've simply mapped existing
database structures to new domain classes. The data, columns, tables,
and databases are all definitely UTF-8 encoded, and as I said before,
the existing Ruby on Rails app (which attaches to the same db
structure) displays the characters fine.

[dwoods@angelinux ~]$ file --mime /tmp/foo
/tmp/foo: text/plain; charset=utf-8
[dwoods@angelinux ~]$

As an interesting side note, when I use just straight Groovy SQL to
pull the data, it too produces a String with 8859_1 character
encoding.

Here's my datasource configuration for my grails app -- maybe
something worthwhile here?

dataSource {
        pooled = true
        driverClassName = "com.mysql.jdbc.Driver"
        username = "xxx"
        password = "xxx"
        dbCreate = "update"
        url = "jdbc:mysql://localhost/database?tcpKeepAlive=true&useUnicode=yes&characterEncoding=UTF-8"
        autoReconnect = true
        dialect = org.hibernate.dialect.MySQL5Dialect
        properties {
                maxActive = 50
                maxIdle = 25
                minIdle = 5
                initialSize = 5
                minEvictableIdleTimeMillis = 60000
                timeBetweenEvictionRunsMillis = 60000
                maxWait = 10000
                validationQuery = "/* ping */"
        }
}

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

If you create a new app and use other db, it doesn't generate the proper characters?

On Jul 13, 2011, at 12:09 AM, Daniel Woods wrote:

>> I think the problem is between the dump and the import of the data (or the data per si, there's something called signature that i don't know what it is), have you tried with a fresh app (no data import) to see if it can generate the proper characters? After you dumped the file have you checked it? eg: file --mime myfile.sql
>>
>
> I'm not sure what "signature" is either as it pertains to character
> encoding. I've not done any data import; I've simply mapped existing
> database structures to new domain classes. The data, columns, tables,
> and databases are all definitely UTF-8 encoded, and as I said before,
> the existing Ruby on Rails app (which attaches to the same db
> structure) displays the characters fine.
>
> [dwoods@angelinux ~]$ file --mime /tmp/foo
> /tmp/foo: text/plain; charset=utf-8
> [dwoods@angelinux ~]$
>
> As an interesting side note, when I use just straight Groovy SQL to
> pull the data, it too produces a String with 8859_1 character
> encoding.
>
> Here's my datasource configuration for my grails app -- maybe
> something worthwhile here?
>
> dataSource {
> pooled = true
> driverClassName = "com.mysql.jdbc.Driver"
> username = "xxx"
> password = "xxx"
> dbCreate = "update"
> url = "jdbc:mysql://localhost/database?tcpKeepAlive=true&useUnicode=yes&characterEncoding=UTF-8"
> autoReconnect = true
> dialect = org.hibernate.dialect.MySQL5Dialect
> properties {
> maxActive = 50
> maxIdle = 25
> minIdle = 5
> initialSize = 5
> minEvictableIdleTimeMillis = 60000
> timeBetweenEvictionRunsMillis = 60000
> maxWait = 10000
> validationQuery = "/* ping */"
> }
> }
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Daniel Woods]

> If you create a new app and use other db, it doesn't generate the proper characters?
>

Same effect. I've also written a small Perl/Dancer app just to test
another language/framework, and it too displays the characters and
character encoding correctly. Groovy/Grails seems to be the only
technology that is having a problem with this :-(

/facepalm

..dan

> On Jul 13, 2011, at 12:09 AM, Daniel Woods wrote:
>
>>> I think the problem is between the dump and the import of the data (or the data per si, there's something called signature that i don't know what it is), have you tried with a fresh app (no data import) to see if it can generate the proper characters? After you dumped the file have you checked it? eg: file --mime myfile.sql
>>>
>>
>> I'm not sure what "signature" is either as it pertains to character
>> encoding. I've not done any data import; I've simply mapped existing
>> database structures to new domain classes. The data, columns, tables,
>> and databases are all definitely UTF-8 encoded, and as I said before,
>> the existing Ruby on Rails app (which attaches to the same db
>> structure) displays the characters fine.
>>
>> [dwoods@angelinux ~]$ file --mime /tmp/foo
>> /tmp/foo: text/plain; charset=utf-8
>> [dwoods@angelinux ~]$
>>
>> As an interesting side note, when I use just straight Groovy SQL to
>> pull the data, it too produces a String with 8859_1 character
>> encoding.
>>
>> Here's my datasource configuration for my grails app -- maybe
>> something worthwhile here?
>>
>> dataSource {
>>       pooled = true
>>       driverClassName = "com.mysql.jdbc.Driver"
>>       username = "xxx"
>>       password = "xxx"
>>       dbCreate = "update"
>>       url = "jdbc:mysql://localhost/database?tcpKeepAlive=true&useUnicode=yes&characterEncoding=UTF-8"
>>       autoReconnect = true
>>       dialect = org.hibernate.dialect.MySQL5Dialect
>>       properties {
>>               maxActive = 50
>>               maxIdle = 25
>>               minIdle = 5
>>               initialSize = 5
>>               minEvictableIdleTimeMillis = 60000
>>               timeBetweenEvictionRunsMillis = 60000
>>               maxWait = 10000
>>               validationQuery = "/* ping */"
>>       }
>> }
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>    http://xircles.codehaus.org/manage_email
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Daniel Woods]

FWIW, what I did was go through and use groovy's
StreamingMarkupBuilder to convert all the abnormal characters to HTML
codes. This will suffice for now, but I'm still kind of annoyed that I
couldn't get it to work with properly encoded UTF-8 characters.

Oh well...

..dan

On Tue, Jul 12, 2011 at 1:07 PM, Daniel Woods <[hidden email]> wrote:

>> If you create a new app and use other db, it doesn't generate the proper characters?
>>
>
> Same effect. I've also written a small Perl/Dancer app just to test
> another language/framework, and it too displays the characters and
> character encoding correctly. Groovy/Grails seems to be the only
> technology that is having a problem with this :-(
>
> /facepalm
>
> ..dan
>
>> On Jul 13, 2011, at 12:09 AM, Daniel Woods wrote:
>>
>>>> I think the problem is between the dump and the import of the data (or the data per si, there's something called signature that i don't know what it is), have you tried with a fresh app (no data import) to see if it can generate the proper characters? After you dumped the file have you checked it? eg: file --mime myfile.sql
>>>>
>>>
>>> I'm not sure what "signature" is either as it pertains to character
>>> encoding. I've not done any data import; I've simply mapped existing
>>> database structures to new domain classes. The data, columns, tables,
>>> and databases are all definitely UTF-8 encoded, and as I said before,
>>> the existing Ruby on Rails app (which attaches to the same db
>>> structure) displays the characters fine.
>>>
>>> [dwoods@angelinux ~]$ file --mime /tmp/foo
>>> /tmp/foo: text/plain; charset=utf-8
>>> [dwoods@angelinux ~]$
>>>
>>> As an interesting side note, when I use just straight Groovy SQL to
>>> pull the data, it too produces a String with 8859_1 character
>>> encoding.
>>>
>>> Here's my datasource configuration for my grails app -- maybe
>>> something worthwhile here?
>>>
>>> dataSource {
>>>       pooled = true
>>>       driverClassName = "com.mysql.jdbc.Driver"
>>>       username = "xxx"
>>>       password = "xxx"
>>>       dbCreate = "update"
>>>       url = "jdbc:mysql://localhost/database?tcpKeepAlive=true&useUnicode=yes&characterEncoding=UTF-8"
>>>       autoReconnect = true
>>>       dialect = org.hibernate.dialect.MySQL5Dialect
>>>       properties {
>>>               maxActive = 50
>>>               maxIdle = 25
>>>               minIdle = 5
>>>               initialSize = 5
>>>               minEvictableIdleTimeMillis = 60000
>>>               timeBetweenEvictionRunsMillis = 60000
>>>               maxWait = 10000
>>>               validationQuery = "/* ping */"
>>>       }
>>> }
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe from this list, please visit:
>>>
>>>    http://xircles.codehaus.org/manage_email
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>    http://xircles.codehaus.org/manage_email
>>
>>
>>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Daniel Henrique Alves Lima]

        Daniel, are you saying that you've generated a *new* Grails
app and the UTF-8 characters were not properly displayed? Can you
share this app?
        If you use HSQLDB (instead of MySQL) with your *new* Grails
app, how does the text look like?
        I know nothing about MySQL, but reading your posts I would bet
there is something weird in your environment or in your use of the
MySQL JDBC driver.

        Grails has some issues rendering ISO-8859-1, even when
everything is configured correctly, but I haven't seen anything like
this for UTF-8 (I'm assuming that every configuration is ok and every
source file is correctly encoded).


        Best regards,

                Daniel.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Daniel Woods]

>        Daniel, are you saying that you've generated a *new* Grails
> app and the UTF-8 characters were not properly displayed? Can you
> share this app?

Yes; It would take me a lot of work to make the application
not-confidential, and since I have a suitable workaround (and that I
agree with you that my environment is doing something weird -- I just
can't figure it out) it's probably not worth the effort for me to do
this. I will tell you though that I took and existing database with
UTF-8 encoded characters, which display correctly in other frameworks,
and mapped my domain layer on top of that to start, and all the
columns that displayed fine in the other frameworks displayed only in
8859-1 from Grails. I also tried migrating the data to a new schema
generated from my domain classes through a Groovy script using
groovy.sql.Sql with hand-written insert statements, and Groovy SQL
didn't preserve the UTF-8 encoding either.

>        If you use HSQLDB (instead of MySQL) with your *new* Grails
> app, how does the text look like?

This I didn't try.

>        I know nothing about MySQL, but reading your posts I would bet
> there is something weird in your environment or in your use of the
> MySQL JDBC driver.

Agreed... I'm just not sure what. From what I can see, I've done
everything correctly.

>
>        Grails has some issues rendering ISO-8859-1, even when
> everything is configured correctly, but I haven't seen anything like
> this for UTF-8 (I'm assuming that every configuration is ok and every
> source file is correctly encoded).
>

No doubt. And all the pages are actually rendering with UTF-8
encoding, but the character set that is displayed is ISO-8859-1. I
confirmed this by using one of the many online UTF-8 to Latin1
converters to convert the proper UTF-8 character to Latin1 format, and
the result was what the GSPs were rendering.

So, in short, I'm sure that it's something that I did wrong along the
way, but even using the existing data that works from Perl and Rails,
I couldn't get Grails or Groovy to render the characters correctly...
A suitable solution was to transform the UTF-8 characters to HTML
codes and store them as such in the database, which is what the Grails
app will do going forward anyway, so I'm sure this will serve as a
long-term solution as well.

Thanks for the email and the feedback.

..dan

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Daniel Henrique Alves Lima]

        Daniel, I'm not asking for your original/main/true/real app.
I'm just asking for a simple (and **new**) Grails app that only
selects one encoded text field in one of your tables. I think you
won't even need to sent it to us.

        Try this:

        1. Create a new app using grails create-app;
        2. Create one domain class that refers one of your tables.
Because you will analyze just one field, I believe you can simply map
one column/property;
        3. Create a controller that performs a
YourDomainObject.withTransaction{YourDomainObject.findAll()};
        4. Write a simple gsp that produce a html using the values in
YourDomainObject.findAll();
        5. Check the results.

        Are they wrongly encoded?

        Best regards,

                Daniel.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[pledbrook]

> Here's my datasource configuration for my grails app -- maybe
> something worthwhile here?
>
> dataSource {
          ...
>        dbCreate = "update"
>        url = "jdbc:mysql://localhost/database?tcpKeepAlive=true&useUnicode=yes&characterEncoding=UTF-8"
>        autoReconnect = true
>        dialect = org.hibernate.dialect.MySQL5Dialect
          ...

One other option: restore the database to ensure all data is UTF-8,
then remove the 'dbCreate' and 'dialect' properties before running
your Grails application again.

Peter

--
Peter Ledbrook
Grails Advocate
SpringSource - A Division of VMware

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Al Baker]

How are you editing your GSPs and other files?

Eclipse, STS?  Make sure you have UTF-8 file encodings of your GSPs and message files and such too. 

What OS are you on?

On Wed, Jul 13, 2011 at 6:39 AM, Peter Ledbrook <[hidden email]> wrote:

> Here's my datasource configuration for my grails app -- maybe
> something worthwhile here?
>
> dataSource {

         ...

>        dbCreate = "update"
>        url = "jdbc:mysql://localhost/database?tcpKeepAlive=true&useUnicode=yes&characterEncoding=UTF-8"
>        autoReconnect = true
>        dialect = org.hibernate.dialect.MySQL5Dialect

         ...

One other option: restore the database to ensure all data is UTF-8,
then remove the 'dbCreate' and 'dialect' properties before running
your Grails application again.

Peter

--
Peter Ledbrook
Grails Advocate
SpringSource - A Division of VMware

---------------------------------------------------------------------

To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email

[Daniel Woods]

>
>        1. Create a new app using grails create-app;

Done. Call it "utf8test".

>        2. Create one domain class that refers one of your tables.
> Because you will analyze just one field, I believe you can simply map
> one column/property;

Done. class Products { String title; String description; static
mapping = { version false } }

>        3. Create a controller that performs a
> YourDomainObject.withTransaction{YourDomainObject.findAll()};

grails generate-all

>        4. Write a simple gsp that produce a html using the values in
> YourDomainObject.findAll();

see #3

>        5. Check the results.
>
>        Are they wrongly encoded?
>

Yes, still.

It's a mystery to me... Thanks for the ideas.

..dan

>        Best regards,
>
>                Daniel.
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Daniel Woods]

>
> One other option: restore the database to ensure all data is UTF-8,
> then remove the 'dbCreate' and 'dialect' properties before running
> your Grails application again.

Done; same effect.

..dan

>
> Peter
>
> --
> Peter Ledbrook
> Grails Advocate
> SpringSource - A Division of VMware
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Daniel Woods]

> How are you editing your GSPs and other files?
>

STS.

> Eclipse, STS?  Make sure you have UTF-8 file encodings of your GSPs and
> message files and such too.
>

STS; I'm not sure what you mean here... They unrendered GSPs have only
ASCII text in them and therefore the interpreted charset comes as
us-ascii. I'm unaware of a way to "force" UTF-8 encoding on a text
file. Please correct me if I'm wrong here.

For example:

[dwoods@angelinux products]$ file --mime list.gsp
list.gsp: text/plain; charset=us-ascii
[dwoods@angelinux products]$ convmv -f ascii -t UTF-8 --nosmart
--notest list.gsp
Ready!
[dwoods@angelinux products]$ file --mime list.gsp
list.gsp: text/plain; charset=us-ascii
[dwoods@angelinux products]$

> What OS are you on?

Fedora 15

I have no idea what the problem is, but my workaround has satisfied
the client, and new data going into the db is being converted to HTML
codes, so for now I'm happy. I'm still curious about this though...
:-/

..dan

>
>
> On Wed, Jul 13, 2011 at 6:39 AM, Peter Ledbrook <[hidden email]>
> wrote:
>>
>> > Here's my datasource configuration for my grails app -- maybe
>> > something worthwhile here?
>> >
>> > dataSource {
>>          ...
>> >        dbCreate = "update"
>> >        url =
>> > "jdbc:mysql://localhost/database?tcpKeepAlive=true&useUnicode=yes&characterEncoding=UTF-8"
>> >        autoReconnect = true
>> >        dialect = org.hibernate.dialect.MySQL5Dialect
>>          ...
>>
>> One other option: restore the database to ensure all data is UTF-8,
>> then remove the 'dbCreate' and 'dialect' properties before running
>> your Grails application again.
>>
>> Peter
>>
>> --
>> Peter Ledbrook
>> Grails Advocate
>> SpringSource - A Division of VMware
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>    http://xircles.codehaus.org/manage_email
>>
>>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[Al Baker]

Unicode is one of those things that I think we'd want to get to the bottom of, even if it's a corner case.  The internals of the JVM are UTF-16, and most modern APIs, formats, and interfaces are UTF-8.  It sounds like we've covered all of the main APIs from the domain model down - the database, the JDBC driver, etc. 

Let's look at the APIs going up to the presentation tier - the GSP, any servlet filters, tag libs, or other things that are having to write information to an output stream.

Can you make sure that STS saves the GSPs in UTF-8? 

Window->Preferences... and Opened General->Workspace and changed the file encoding setting to UTF-8

Also in 'gedit' on Linux, the 'save as' dialog has encoding in it as well. You could dump your database to a flat file, and see if it comes out as UTF-8 using gedit.

Al

On Wed, Jul 13, 2011 at 3:36 PM, Daniel Woods <[hidden email]> wrote:

> How are you editing your GSPs and other files?
>

STS.

> Eclipse, STS?  Make sure you have UTF-8 file encodings of your GSPs and
> message files and such too.
>

STS; I'm not sure what you mean here... They unrendered GSPs have only
ASCII text in them and therefore the interpreted charset comes as
us-ascii. I'm unaware of a way to "force" UTF-8 encoding on a text
file. Please correct me if I'm wrong here.

For example:

[dwoods@angelinux products]$ file --mime list.gsp
list.gsp: text/plain; charset=us-ascii
[dwoods@angelinux products]$ convmv -f ascii -t UTF-8 --nosmart
--notest list.gsp
Ready!
[dwoods@angelinux products]$ file --mime list.gsp
list.gsp: text/plain; charset=us-ascii
[dwoods@angelinux products]$

> What OS are you on?

Fedora 15

I have no idea what the problem is, but my workaround has satisfied
the client, and new data going into the db is being converted to HTML
codes, so for now I'm happy. I'm still curious about this though...
:-/

..dan

>
>
> On Wed, Jul 13, 2011 at 6:39 AM, Peter Ledbrook <[hidden email]>
> wrote:
>>
>> > Here's my datasource configuration for my grails app -- maybe
>> > something worthwhile here?
>> >
>> > dataSource {
>>          ...
>> >        dbCreate = "update"
>> >        url =
>> > "jdbc:mysql://localhost/database?tcpKeepAlive=true&useUnicode=yes&characterEncoding=UTF-8"
>> >        autoReconnect = true
>> >        dialect = org.hibernate.dialect.MySQL5Dialect
>>          ...
>>
>> One other option: restore the database to ensure all data is UTF-8,
>> then remove the 'dbCreate' and 'dialect' properties before running
>> your Grails application again.
>>
>> Peter
>>
>> --
>> Peter Ledbrook
>> Grails Advocate
>> SpringSource - A Division of VMware
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this list, please visit:
>>
>>    http://xircles.codehaus.org/manage_email
>>
>>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email

[Daniel Henrique Alves Lima]

    I don't think this is a mystery :-)

    Using your test results, we can conclude that

    1. Your db is wrongly encoded;
    2. Your JDBC driver is unable to properly convert the encoded data;

     Can you answer the following questions?

    1. Which OS are you using? If you're using some *nix, execute

    uname -a

    2. Which JRE/JDK are you using?

    which java ; java -version ; echo $JAVA_HOME

    3. Which version of Grails are you using?

    which grails ; grails ; echo $GRAILS_HOME

    4. What are the default language and encoding in use?

    echo $LANG

    5. Are you changing the default language or encoding of your JVM?

    Add

    println System.properties

    to your BootStrap.groovy and send the results to us (if you can)

    6. Which version of MySQL are you using?


    7. Which version of the MySQL JDBC driver are you using?
   

    8. Can you share the test application that you've created? Just
replace the database user/password/server and send it to us (or to me in
private), if you can.



Daniel Woods wrote:

>>        1. Create a new app using grails create-app;
>>    
>
> Done. Call it "utf8test".
>
>  
>>        2. Create one domain class that refers one of your tables.
>> Because you will analyze just one field, I believe you can simply map
>> one column/property;
>>    
>
> Done. class Products { String title; String description; static
> mapping = { version false } }
>
>  
>>        3. Create a controller that performs a
>> YourDomainObject.withTransaction{YourDomainObject.findAll()};
>>    
>
> grails generate-all
>
>  
>>        4. Write a simple gsp that produce a html using the values in
>> YourDomainObject.findAll();
>>    
>
> see #3
>
>  
>>        5. Check the results.
>>
>>        Are they wrongly encoded?
>>
>>    
>
> Yes, still.
>
> It's a mystery to me... Thanks for the ideas.
>
> ..dan
>
>
>  
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

[jirkabacovsky]

If you are using eclipse for your project check your "text file encoding" (window-preferences-general-workspace-Text file encoding)
I had similar problem. After changing it to UTF-8 it works fine.