Hacking together an ABAP Syntax-Highlighter : Extending Prism.js

Provoked by this post: "SCN Syntax Highlighting for Dummies on SCN", I started looking into the possibility of hacking together a syntax-highlighter for ABAP – preferably a JavaScript library that can highlight the ABAP code at the time of web page load on the client side/Browser.

Way back in 2004/5, remember when we were young :) - I was putting together my first personal website www.geocities.com/rmtiwari with the purpose of sharing ABAP code and world domination :), of course. That site was long abandoned but still exist as a reminder of past at www.passionateaboutsap.com . At that time, I wrote an ABAP utility program, to download ABAP reports/functions as HTML page and *potentially* make the ABAP code presentable and readable on Web. I thought it was cool at that time – specially using html anchors to move from a form-routine or variable use to the declaration part, while reading code on web. It was not perfect or easy to use and I did not use it much myself :).

Anyway, with current ABAP editor, you can actually export the highlighted code in HTML/RTF/PDF. I was not aware of this till I started looking into it a few days back. You can assign a quick-access key as below and then it’s easy to get the code downloaded from editor in these formats. If a website/blog does not provide auto-syntax-highlighting of ABAP code and you are a recovering perfectionist then this could be one option - but it’s not what I am looking for.Now a days Browsers are a lot more powerful so performance of client side processing is not such an issue - and most of the languages have open-source JavaScript libraries for on-the-fly syntax highlighting on the client side.

As an ABAP Programmer, it was a bit embarrassing to read that SAP developers will have to go through so many manual steps, as mentioned in that Post - just to get a decent highlighting of ABAP code on SCN. But my Blog does not have any syntax highlighting for ABAP either…and that’s why it was a bit embarrassing to realize that even on my Blog, all other language/formats code (e.g. JavaScript, XML, HTML etc. ) has or can have the syntax-highlighting except for my main Programming Language – ABAP.

I must admit that I’ve not tried some of the online options, mentioned in the comments section of that post. I guess there is no harm in re-inventing the wheel – who knows it might just still be stone-age in progress :) - and I do have a few hours at my disposal.

Requirement:

So basically, I wanted to put together a JavaScript library that can achieve this syntax-highlighting effectively – I was looking for a practical solution with acceptable margins of error ( It’s not hugely critical even if it does not work for a code-line or two ) but something that I can hack together using my modest JavaScript skills and within a few hours of one-time effort, deploy it on my Blog within a few mins and when I write a post with code, I won’t like to spend more than a few seconds to get the highlighting working on the code-part. Page load Performance is …well important too. So with these vague functional and non-functional requirements and constraints, I was off to Google.

Exploration:

While searching on web for any existing such library, I found quite a few JavaScript libraries but none with an ABAP extension. There are a few other website/libraries that offer syntax-highlighting for ABAP, but most of these require some server side components in e.g. PHP / Python etc. After a quick review of my search results, I decided to use Prism and downloaded the SQL Language highlighter to start as a base for ABAP extension.

Solution:

With the prism download, you get prism.js and prism.css for the particular programming language - in our case SQL.
Now to extend it for ABAP, the idea is to just change the definition of programming language – e.g. adapt the Regex Patterns for comments, strings and keywords etc.
Looking at the programming definition part of SQL language in prism.js , the definition was as below:

So now the complex bit was to change the language definition to match ABAP syntax for comments, strings, keywords etc. After searching through the web to find Regex patterns for related purpose, tinkering the Regex patterns ( playing on refiddle ) and quite a few iterations, I finally managed the following definition that seems to work reasonably well – though there are several shortcomings but none are life-threatening, I guess .

Quick explanation of Language-definition:

• Comment: Whole line comment in ABAP start with asterisk (*) at the beginning of the line. That was relatively easier to implement as Regex pattern: /^[\*].*$/gm,
• Comment1: ( should have named properly ): A double quotation mark (") can also be used to add a comment and part of the line after " will become commented . Initially, I tried handling both type of comments with one pattern but that was creating a problem with strings in code - strings can contain double quotation marks but should not be shown as commented line. So I spilt the comments definition Regex in two parts, * type of comments will be evaluated first then the strings and finally the in-line comments.
• String:Hardcoded Strings/characters in code can be quoted within ` `or ' '.
• Keywords: In order to get the ABAP Keywords, I used the same XML that is also used by the SAP standard front-end editor ( I think ). These are ABAP Editor’s XML files in SAPGUI working directory. Using Notepad/MSWord, I concatenated these keywords to a pipe separated string in descending order of keyword-width – so that keywords like “no-gap” will be matched first and “no” will be matched later – otherwise only “no” will get highlighted in “no-gap”. Also I noticed that the XML file does not contain all the keywords – added a few like TRANSFORMATION that were missing - so I may need to refresh the front-end XMLs to get the latest version somehow – TBD.

There are still some obvious issues and there will be many more that I can’t think of yet. For example, in case the in-line comment contains double quoted characters itself. Further, some of the keywords can appear in structure component/variable names declaration part etc. I did manage to avoid the highlighting of structure components having a keyword as name, by making slight change in the main JavaScript code, but this will not work for data-declaration part.

How to use ABAP syntax-highlighter on a website / blog:

• Site/Blog Administrator/owner should add the JavaScript and CSS files as below (both files can be downloaded from the links). Obviously this is one time activity.
  

  <head>
  ....
  <link href='http://www.pragmatiqa.com/docs/ABAPSyntaxHighlighter/Final/prismABAP.css' rel='stylesheet'/>
  <script src='http://www.pragmatiqa.com/docs/ABAPSyntaxHighlighter/Final/prismABAP.js'></script>
  ....
  </head>
  
  

• Post writer will enclose the ABAP code with <pre> and <code> and specify the CSS class name: line-numbers language-abap as below:

<pre class="line-numbers language-abap"><code>
* Actual ABAP Code lines should be pasted here  – site/blog post editor should take care of converting any starting html brackets in ABAP code to &lt; for example field-symbols etc.
</code></pre>

• A sample highlighted code is as below and also I’ve updated my older posts to use this for ABAP code highlighting now. I’ve not checked the browser / page-load performance yet but looks like I am going to use this on my Blog.
  If you want to try on your Blog/site – it’s free unless you are hell-bent on paying for it :)
  
  

   REPORT  z_ram_test_csv_to_itab_regex.
  * this is a comment by ramtiwari.
  * select '100' from something or just multiply to check that * does not comment
  
   a = 1 * 10.
  
  DATA : g_input TYPE string.
  DATA : l_tab TYPE c value cl_abap_char_utilities=>HORIZONTAL_TAB.
  DATA : l_regex type string.
  
  g_input = '"F1",F2,""Hello",  "How" are "you"","F3"'.
  
  REPLACE ALL OCCURRENCES OF
    REGEX ',(?=(?:[^"]*$)|(?:[^"]*"[^"]*"[^"]*)*$)'
    IN g_input WITH l_tab.
  
  REPLACE FIRST OCCURRENCE OF
    REGEX '^"'
    IN g_input WITH ' '.
  
  CONCATENATE '"' l_tab '"|"' l_tab '|' l_tab '"' into l_regex.
  REPLACE ALL OCCURRENCES OF
    REGEX l_regex                         "    '"l_tab"|"l_tab|l_tab"'
    IN g_input WITH l_tab.
  
  REPLACE ALL OCCURRENCES OF '""' IN g_input
    WITH '"'.    "Added during testing
  
  REPLACE ALL OCCURRENCES OF REGEX '"$' IN g_input
    WITH ' '.
  
  WRITE :/ g_input.
  
  
  

Next..

This post was a slight distraction from my original blog-post-series related to Netweaver Gateway OData Services but I am currently hacking together a web-based, slightly-interactive UI, to graphically show the data-model of related tables (during prototyping stage of service development) or of an already developed Service. It might be useful to get Functional Consultant, ABAP Developer and External UI developer on the same page during OData Service Prototyping/verification and possibly save the innocents (e.g. Functional Guys) from the so called - death by XML or Eclipse. I am playing with JavaScript libraries, datajs for OData-metadata-xml parsing, a layouting library dagre to get the position of the nodes and jsPlumb / JQueryUI for Graph/UI rendering. Initial results are as below (connected draggable boxes to show the entity/properties and relationships):