Help
Join the Community
-
Documentation
Find detailed info about ServiceNow products, apps, features, and releases.
-
Impact
Drive a faster ROI and amplify your expertise with ServiceNow Impact.
-
Partner
Grow your business with promotions, news, and marketing tools for partners.
-
Store
Download certified apps and integrations that complement ServiceNow.
After web services were first introduced, they became very popular. Web services is a great tool to exchange information. ServiceNow implementations of the web services are top notch. When encoding international characters, the safest option points to Unicode. One of the most popular implementations is UTF-8, which is the one adopted by us. If you need to connect to your instance and use non-ascii characters, you should read this blog. Specially if you are seeing � or questions marks (?) as part of the data received using web services. Luckily the solution is very simple by safe-encoding non-ascii characters before sending them (see below) if you are not using UTF-8.
Outside encoding, incoming SOAP requests with characters like "&", "<" or ">" on the data can cause errors. Those characters are reserved as they are part of the XML tags used to delimiter the data. If used on the data, they interfere with the SOAP message itself. If they are passed on the data and read on ServiceNow, they will show as "Unable to parse SOAP document"
Recommendations to safely pass non-ascii characters
I will focus on SOAP web services. They use SOAP messages to exchange data between the soap nodes (e.g. your soap client and the instance). However, while the data is provided on the message, both parties need to agree on the "encoding." We make it easy, we would use UTF-8 to encode characters.
If you are interacting with your instance and you use a different encoding with special characters on your data, you may face problems if you are not using Unicode. However, XML offers the option to safe encoding most characters into escape characters (safe encoded).
For SOAP Message | If the message is encoded on UTF-8 | Recommendation: The message needs to be safe encoded |
Incoming to the instance | Yes | No, it is not necessary. If safe encoded, it will also work. |
Incoming to the instance | No | Yes, safe encode the data to avoid ? or � characters |
Outbound from the instance | Yes | Yes, if target is not UTF-8 to avoid ? or � characters |
Outbound from the instance | No | Yes, always safe encode the data |
The outbound soap messages from the instance are always on UTF-8, so the messages are always encoded on UTF-8
"Incoming" means that SOAP calls toward your instance. "Outbound" means SOAP calls from the instance to your end-point. To safely encode the message, you need to transform any non-ascii character into a XML code.
Example of a message sent to the instance using an Unicode (UTF-8) encoding
On the following example, I will use SOAP UI, to transfer "comments" that contains non-ascii characters and "work_notes" that contains the same characters safely encoded. I would expect the system will have NO problem with the characters as both soap nodes are using UTF-8.
Using SOAP UI looks like:
In more detail, the message looks like:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:inc="http://www.service-now.com/incident">
<soapenv:Header/>
<soapenv:Body>
<inc:insert>
<short_description>Testing with encoding characters characters</short_description>
<comments>Testing with encoding characters characters -not safe encoded
Basic Latin
! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z [ \ ] ^ _ ` a b c d e f g h i j k l m n o p q r s t u v w x y z { | } ~
Latin-1 Supplement
¡ ¢ £ ¤ ¥ ¦ § ¨ © ª « ¬ ® ¯ ° ± ² ³ ´ µ ¶ · ¸ ¹ º » ¼ ½ ¾ ¿ í€ í� Â íƒ í„ í… í† í‡ íˆ É íŠ í‹ íŒ í� íŽ í� í� í‘ í’ í“ í” í• í– × í˜ í™ íš í› íœ í� íž ß í á â ã ä å æ ç è é ê í« ì í î í¯ í° ñ ò ó ô õ ö í· ø ù ú û ü ý í¾ í¿
Latin Extended-A
Ä€ Đ Ä‚ ă Ä„ Ä… Ć ć Ĉ ĉ ÄŠ Ä‹ ÄŒ Đ ÄŽ Đ Đ đ Ä’ Ä“ Ä” Ä• Ä– Ä— Ę Ä™ Äš Ä› Äœ Đ Äž ÄŸ Ä Ä¡ Ä¢ Ä£ Ĥ Ä¥ Ħ ħ Ĩ Ä© Ī Ä« Ĭ Ä Ä® į Ä° ı IJ ij Ä´ ĵ Ķ Ä· ĸ Ĺ ĺ Ä» ļ Ľ ľ Ä¿ Å€ Ł Å‚ Ń Å„ Å… ņ Ň ň ʼn ÅŠ Å‹ ÅŒ Ł ÅŽ Ł Ł Å‘ Å’ œ Å” Å• Å– Å— Ř Å™ Åš Å› Åœ Ł Åž ÅŸ Å š Å¢ Å£ Ť Å¥ Ŧ ŧ Ũ ũ Ū Å« Ŭ Å Å® ů Å° ű Ų ų Å´ ŵ Ŷ Å· Ÿ Ź ź Å» ż Ž ž Å¿
Latin Extended-B
Æ€ Æ� Æ‚ ƃ Æ„ Æ… Ɔ Ƈ ƈ Ɖ ÆŠ Æ‹ ÆŒ Æ� ÆŽ Æ� Æ� Æ‘ Æ’ Æ“ Æ” Æ• Æ– Æ— Ƙ Æ™ Æš Æ› Æœ Æ� Æž ÆŸ Æ ơ Æ¢ Æ£ Ƥ Æ¥ Ʀ Ƨ ƨ Æ© ƪ Æ« Ƭ Æ Æ® Ư ư Ʊ Ʋ Ƴ Æ´ Ƶ ƶ Æ· Ƹ ƹ ƺ Æ» Ƽ ƽ ƾ Æ¿ Ç€ Ç� Ç‚ ǃ Ç„ Ç… dž LJ Lj lj ÇŠ Ç‹ ÇŒ Ç� ÇŽ Ç� Ç� Ç‘ Ç’ Ç“ Ç” Ç• Ç– Ç— ǘ Ç™ Çš Ç› Çœ Ç� Çž ÇŸ Ç Ç¡ Ç¢ Ç£ Ǥ Ç¥ Ǧ ǧ Ǩ Ç© Ǫ Ç« Ǭ Ç Ç® ǯ Ç° DZ Dz dz Ç´ ǵ Ƕ Ç· Ǹ ǹ Ǻ Ç» Ǽ ǽ Ǿ Ç¿ ...</comments>
</comments>
<work_notes>Testing with encoding characters characters -safe encoded
Basic Latin
! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z [ \ ] ^ _ ` a b c d e f g h i j k l m n o p q r s t u v w x y z { | } ~
Latin-1 Supplement
¡ ¢ £ ¤ ¥ ¦ § ¨ © ª « ¬ ­ ® ¯ ° ± ² ³ ´ µ ¶ · ¸ ¹ º » ¼ ½ ¾ ¿ À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï Ð Ñ Ò Ó Ô Õ Ö × Ø Ù Ú Û Ü Ý Þ ß à á â ã ä å æ ç è é ê ë ì í î ï ð ñ ò ó ô õ ö ÷ ø ù ú û ü ý þ ÿ
Latin Extended-A
Ā ā Ă ă Ą ą Ć ć Ĉ ĉ Ċ ċ Č č Ď ď Đ đ Ē ē Ĕ ĕ Ė ė Ę ę Ě ě Ĝ ĝ Ğ ğ Ġ ġ Ģ ģ Ĥ ĥ Ħ ħ Ĩ ĩ Ī ī Ĭ ĭ Į į İ ı IJ ij Ĵ ĵ Ķ ķ ĸ Ĺ ĺ Ļ ļ Ľ ľ Ŀ ŀ Ł ł Ń ń Ņ ņ Ň ň ʼn Ŋ ŋ Ō ō Ŏ ŏ Ő ő Œ œ Ŕ ŕ Ŗ ŗ Ř ř Ś ś Ŝ ŝ Ş ş Š š Ţ ţ Ť ť Ŧ ŧ Ũ ũ Ū ū Ŭ ŭ Ů ů Ű ű Ų ų Ŵ ŵ Ŷ ŷ Ÿ Ź ź Ż ż Ž ž ſ
Latin Extended-B
ƀ Ɓ Ƃ ƃ Ƅ ƅ Ɔ Ƈ ƈ Ɖ Ɗ Ƌ ƌ ƍ Ǝ Ə Ɛ Ƒ ƒ Ɠ Ɣ ƕ Ɩ Ɨ Ƙ ƙ ƚ ƛ Ɯ Ɲ ƞ Ɵ Ơ ơ Ƣ ƣ Ƥ ƥ Ʀ Ƨ ƨ Ʃ ƪ ƫ Ƭ ƭ Ʈ Ư ư Ʊ Ʋ Ƴ ƴ Ƶ ƶ Ʒ Ƹ ƹ ƺ ƻ Ƽ ƽ ƾ ƿ ǀ ǁ ǂ ǃ DŽ Dž dž LJ Lj lj NJ Nj nj Ǎ ǎ Ǐ ǐ Ǒ ǒ Ǔ ǔ Ǖ ǖ Ǘ ǘ Ǚ ǚ Ǜ ǜ ǝ Ǟ ǟ Ǡ ǡ Ǣ ǣ Ǥ ǥ Ǧ ǧ Ǩ ǩ Ǫ ǫ Ǭ ǭ Ǯ ǯ ǰ DZ Dz dz Ǵ ǵ Ƕ Ƿ Ǹ ǹ Ǻ ǻ Ǽ ǽ Ǿ ǿ
</work_notes>
</inc:insert>
</soapenv:Body>
</soapenv:Envelope>
Once processed into the target table, the characters are correctly displayed.
This shows that the data is processed without any problems.
Example of a message sent to the instance using a non-Unicode (ISO-8859-1) encoding
Similarly to the previous example, I will use SOAP UI, to transfer "comments" that contains non-ascii characters and "work_notes" that contains the same characters safely encoded. This time I will encode on 'iso-8859-1'. I would expect the system will try to match the characters against UTF-8.
Using soap UI looks like:
In more detail looks like:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:inc="http://www.service-now.com/incident">
<soapenv:Header/>
<soapenv:Body>
<inc:insert>
<short_description>Testing with encoding characters characters</short_description>
<comments>Testing with encoding characters characters -not safe encoded
Basic Latin
! " # $ % ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; @ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z [ \ ] ^ _ ` a b c d e f g h i j k l m n o p q r s t u v w x y z { | } ~
Latin Extended-A
A a Ä‚ ă Ä„ Ä… Ć ć C c C c ÄŒ Đ ÄŽ Đ Đ đ E e E e E e Ę Ä™ Äš Ä› G g G g G g G g H h H h I i I i I i I i I i J j K k Ĺ ĺ L l Ľ ľ Ł Å‚ Ń Å„ N n Ň ň O o O o Ł Å‘ O o Å” Å• R r Ř Å™ Åš Å› S s Åž ÅŸ Å š Å¢ Å£ Ť Å¥ T t U u U u U u Å® ů Å° ű U u W w Y y Y Ź ź Å» ż Ž ž
Latin Extended-B
b Đ F f I l O O o t T U u z | ! A a I i O o U u U u U u U u U u A a G g G g K k O o O o j
</comments>
<work_notes>Testing with encoding characters characters -safe encoded
Basic Latin
! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z [ \ ] ^ _ ` a b c d e f g h i j k l m n o p q r s t u v w x y z { | } ~
Latin-1 Supplement
¡ ¢ £ ¤ ¥ ¦ § ¨ © ª « ¬ ­ ® ¯ ° ± ² ³ ´ µ ¶ · ¸ ¹ º » ¼ ½ ¾ ¿ À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï Ð Ñ Ò Ó Ô Õ Ö × Ø Ù Ú Û Ü Ý Þ ß à á â ã ä å æ ç è é ê ë ì í î ï ð ñ ò ó ô õ ö ÷ ø ù ú û ü ý þ ÿ
Latin Extended-A
Ā ā Ă ă Ą ą Ć ć Ĉ ĉ Ċ ċ Č č Ď ď Đ đ Ē ē Ĕ ĕ Ė ė Ę ę Ě ě Ĝ ĝ Ğ ğ Ġ ġ Ģ ģ Ĥ ĥ Ħ ħ Ĩ ĩ Ī ī Ĭ ĭ Į į İ ı IJ ij Ĵ ĵ Ķ ķ ĸ Ĺ ĺ Ļ ļ Ľ ľ Ŀ ŀ Ł ł Ń ń Ņ ņ Ň ň ʼn Ŋ ŋ Ō ō Ŏ ŏ Ő ő Œ œ Ŕ ŕ Ŗ ŗ Ř ř Ś ś Ŝ ŝ Ş ş Š š Ţ ţ Ť ť Ŧ ŧ Ũ ũ Ū ū Ŭ ŭ Ů ů Ű ű Ų ų Ŵ ŵ Ŷ ŷ Ÿ Ź ź Ż ż Ž ž ſ
Latin Extended-B
ƀ Ɓ Ƃ ƃ Ƅ ƅ Ɔ Ƈ ƈ Ɖ Ɗ Ƌ ƌ ƍ Ǝ Ə Ɛ Ƒ ƒ Ɠ Ɣ ƕ Ɩ Ɨ Ƙ ƙ ƚ ƛ Ɯ Ɲ ƞ Ɵ Ơ ơ Ƣ ƣ Ƥ ƥ Ʀ Ƨ ƨ Ʃ ƪ ƫ Ƭ ƭ Ʈ Ư ư Ʊ Ʋ Ƴ ƴ Ƶ ƶ Ʒ Ƹ ƹ ƺ ƻ Ƽ ƽ ƾ ƿ ǀ ǁ ǂ ǃ DŽ Dž dž LJ Lj lj NJ Nj nj Ǎ ǎ Ǐ ǐ Ǒ ǒ Ǔ ǔ Ǖ ǖ Ǘ ǘ Ǚ ǚ Ǜ ǜ ǝ Ǟ ǟ Ǡ ǡ Ǣ ǣ Ǥ ǥ Ǧ ǧ Ǩ ǩ Ǫ ǫ Ǭ ǭ Ǯ ǯ ǰ DZ Dz dz Ǵ ǵ Ƕ Ƿ Ǹ ǹ Ǻ ǻ Ǽ ǽ Ǿ ǿ
</work_notes>
</inc:insert>
</soapenv:Body>
</soapenv:Envelope>
Below, you can see some characters would be translated. However, with XML encoded characters, you can safely send UTF-8 characters.
Using non-ascii characters can cause them to get transferred incorrectly. However, using XML encoded characters you can safely transfer those characters.
Using XML encoded characters, you can safely transfer non-ascii characters when encoding is not UTF-8.
How to safely encode XML data
There are several methods to achieve the data to be encoded. Below is a simple script that can encode the data transferred.
Here is a simple background script function that encode data:
// Simple encoding XML data function - Do not double-escape any characters.
function escapeXMLEntities(xmldata) {
return xmldata.replace(/[\u00A0-\u2666<>\&]/g, function (a) {
return "&#" + a.charCodeAt(0) + ";"
})
};
var str = "A a Ä‚ ă Ä„ Ä… Ć ć C c C c ÄŒ Đ ÄŽ Đ Đ đ E e E e E e Ę Ä™ Äš Ä› G g G g G g G g H h H h I i I i I i I i I i J j K k Ĺ ĺ L l Ľ ľ Ł Å‚ Ń Å„ N n Ň ň O o O o Ł Å‘ O o Å” Å• R r Ř Å™ Åš Å› S s Åž ÅŸ Å š Å¢ Å£ Ť Å¥ T t U u U u U u Å® ů Å° ű U u W w Y y Y Ź ź Å» ż Ž ž";
gs.print(escapeXMLEntities(str));
Result:
Script completed in scope global: script
*** Script: A a Ă ă Ą ą Ć ć C c C c Č č Ď ď Đ đ E e E e E e Ę ę Ě ě G g G g G g G g H h H h I i I i I i I i I i J j K k Ĺ ĺ L l Ľ ľ Ł ł Ń ń N n Ň ň O o O o Ő ő O o Ŕ ŕ R r Ř ř Ś ś S s Ş ş Š š Ţ ţ Ť ť T t U u U u U u Ů ů Ű ű U u W w Y y Y Ź ź Ż ż Ž ž
When using SOAP messages, ensure you are using UTF-8 to transfer the data, or ensure non-ascii characters are escaped safely into XML safe codes. A good note, this can also be used on many other situations. Luckily, programmatically, it is not a big challenge.
More information can be found here:
- SOAP web service
- Unable to specify character encoding "UTF-8" for JDBC connection to SQL server (KB0538864)https://hi.service-now.com/kb_view.do?sysparm_article=KB0550868
- Data imported using an outbound SOAP function via a MID server displays two-byte utf-8 characters in...https://hi.service-now.com/kb_view.do?sysparm_article=KB0621602
- Four-byte UTF-8 emojis cause email to be truncated in database (KB0621602)
- Unable to specify character encoding "UTF-8" for JDBC connection to SQL server (KB0538864)
- UTF-8 - Wikipedia
- List of XML and HTML character entity references - Wikipedia
- Web services
- Character encoding - Wikipedia
- My other blogs
3 Comments
