MetaL XML based Meta-Programming Language PHP Conference 2001 Frankfurt - Germany, November 5-7, 2001 Manuel Lemos c 2001 mlemos@acm.org October 28, 2001 1 1. History This part tells the history of the creation and development of MetaL. 2 1.1 The Web ERP project Late 1998: The Web ERP project was started as a proposal of some investors. The goal was to develop a Web based business management suite of programs: accounting, billing, stocks, etc.. The project was taking too long with only one person learning the business processes, planning and developing. Finding available qualied Web developers was hard and when you found them they were expensive to hire due to the Internet hype. | 3 1.2 The birth of MetaL Middle 1999: MetaL was born with special goals meant to defeat these diculties: Æ Be simple enough to not require expensive qualied developers to write applications with it. Æ Allow the design of business domain specic languages to help writing applications faster. Late 1999: The original investors back out due to lack of faith on the conclusion in useful time of any part of the project that had market value. | 4 1.3 The Integral project Early 2000: A previous business partner proposes to invest on restarting the Web ERP project. The project was rebaptized as Integral - a generic integrated system for modular Web applications. A few modules were developed: Æ Base installation and setup of modules Æ User access management Æ Companies records management Late 2000: The Integral project was suspended due to an irrefusable job proposal. | 5 1.4 Today Middle 2001: MetaL was opened to closed circle of trusted developers for evaluation in preparation for release as Open Source. MetaL was presented publically for the rst time in the O'Reilly Open Source Conference (http://conferences. oreillynet.com/cs/os2001/view/e_sess/1874) in San Diego. MetaL and Integral are expected to be released as Open Source projects until the end of 2001. The development of MetaL and the Integral project are being carried out as my free time permits. || 6 2. Motivation This part details the reasons that motivated the development of a Meta-Programming Language. 7 2.1. The ideal language Question: What is the best programming language? Answer: It depends on your purpose. Question: What if there was a language that is suitable for all purposes? Answer: There is no such ideal language. Question: What if there was a language that could be translated into other languages? Answer: That is the main goal of MetaL: a meta-programming language. | 8 2.2. Meta-Programming Developing a new language from scratch would be of great risk and would take much time. Giving up on existing languages would mean giving up on their maturity and tness to solve real world problems. Developing a meta-programming compiler would represent a compromise between developing a new language without giving up on existing languages. | 9 2.3. Future opportunities The adoption of MetaL as the language for software development may present opportunities that developers may benet in the future. Æ Retargettable Programming If I can generate a program in dierent programming languages, I may as well generate its code in a suitable manner to run optimally on dierent platforms, even those that do not exist today. Æ Future-proof Software The languages that are most appropriate for developing applications today, may not be the best choice for developing future applications. Whatever are the languages of choice in the future, MetaL programs and components may not need to be 10 adapted to be generated in the languages of the future. | 11 2.4. Language of powerful commands Programs written in a language with powerful commands need less code to write. Æ Less development time -> Less time to market Æ Less bugs -> Higher code quality Æ Greater code maintainability -> Less time to develop new features -> Greater user satisfation | 12 2.5. Domain specic languages A programming tool that is appealing to less qualied developers, may also be appealing to experts on areas that are important for a software product. For instance: Æ An economist could dene business rules to be implemented by a software product. Æ A translator could localize a software product. These experts could participate directly in the development of a software product, especially if they do not have to learn traditional programming languages. Domain specic languages could be used by these professionals. Domain experts that are not programmers, are easier to nd and less expensive to hire. | 13 2.6. Meta-Meta-Programming If a meta-programming language can be used to generate programs in any target language, it may as well be used to generate programs in the meta-language itself. This means that there may be very high level commands in a meta-language that can be interpreted to generate source code with only lower level meta-language commands. Translation of higher level commands in lower level commands may be done in one or more compiler passes. In a nal pass, the meta-programming compiler generates code in a given target language. The method of developing programs with meta-language commands that are translated into lower level commands is called Meta-Meta-Programming or just Meta-N-Programming. Very high level domain specic languages can be developed 14 with more exibility with Meta-Meta-Programming. || 15 3. Introduction to MetaL This part explains how MetaL works. 16 3.1. XML as programming source code Developing a compiler is not a trivial programming matter. It requires knowledge on: Æ Parsing source code with a given syntax. Æ Generating program output in a target executable format. Using XML as source code greatly simplies the problem. Æ XML requires a small set of strict syntax rules. Æ XML parsers are much simpler to build. Æ Generic XML parsers are widely available |. 17 3.2. Programming as simple as cooking People learn faster abstract concepts when these are presented using analogies with real world concepts. Not everybody is able to cook or only eats cooked food, but... Cooking is a real world concept that everybody understands how it works. 1. First, you need to nd and prepare the ingredients. 2. Then, you follow the recipe steps. | 18 3.3. MetaL ingredients The ingredients are the commands of the language that you want to use. Each command is associated with a XML tag. MetaL commands are dened in input les. | 19 3.4. MetaL recipe steps The recipe steps are just a sequence of XML tags that dene your program commands. MetaL commands are dened in output les. | 20 3.5. Compiling MetaL programs 1. To build MetaL program, the compiler takes an input le and an output le. 2. The compiler examines the input le to gure the language commands that you want to use. 3. Input les also dene which compiler modules implement each command. 4. The compiler traverses the output le and invokes the commands associated to each XML tag by calling the respective compiler module. | 21 3.6. Generating target source code Each compiler module may execute any type of action to implement a MetaL command. Actions may be immediate, like creating a le, or may be generating some output like the lines of code of a program in a given target language. A compiler module may be able to generate code for the same MetaL command in multiple target languages. The compiler bindings dene which module implements the translation of commands to a given target language. The denition of the compiler bindings is done in a separate XML le with a specic structure. || 22 4. State of development MetaL This part describes the current state of MetaL. 23 4.1. Current implementation The current implementation of MetaL compiler engine and available modules was written in PHP (http://www.php.net/). PHP is a scripting language that was originally meant for Web programming. It may seem odd to the use PHP for a heavy task such as compiling source les. Today PHP is not such a slow language, thanks to Zend (http://www.zend.com/) compiler and optimizer. The MetaL compiler engine also makes extensive use of caching of previously parsed XML les. | 24 4.2. Available modules - Project building All compiler parts and modules are implemented as OOP classes. The compiler modules were not meant just to generating programs source code. Some modules were develop to assist all steps of project building: Æ compiler - Compiler core engine and base services Æ build - Makele-like build process control Æ documentation - Generation of documentation from general a device independent document denition format Æ distribution - Creation of distribution packages from programs built with MetaL | 25 4.3. Available modules - Runtime control The most important meta-programming modules are meant for controlling the execution ow and data manipulation. Æ expressions - Data type manipulation functions and operators Æ flow - Execution logic ow control constructs Æ script - Generation of output source code as scripts. Æ program - Access to context values passed to the programs Æ class - Creation, access and documentation of OOP classes | 26 4.4. Available modules - Web programming Some modules were developed to assist Web programming in specic: Æ cgi - Access to data values interchanged with Web scripts via CGI Æ options - Access to application wide global option values Æ template - Generation of output from text based template les Æ locale - Localization of application texts and data | 27 4.5 Available modules - System resources access The remaining modules are meant for accessing to system resources: Æ file - File and directory access functions Æ database - DBMS independent SQL database access Æ time - Date and time access manipulation functions Æ xml - XML parsing functions using Expat (http://www.jclark.com/xml/expat.html) Æ crypt - Cryptographic functions Æ system - External program invocation functions Æ menu, image, link, form, email - Interface with existing native PHP classes | 28 4.6. Examples of existing applications Integral (http://www. integral.UpperDesign. com/)Modular integrated sys- temof Web based applications. Web based ezmlm mailing listcreation and conguration tool (http://phpclasses. UpperDesign.com/browse. html/package/177) Multi-page forms class (http://phpclasses. UpperDesign.com/browse. html/package/108) Table wrapper base class (http://phpclasses. UpperDesign.com/browse. html/package/120) Calendar generation class (http://phpclasses. UpperDesign.com/browse. html/package/121) Query result table display class (http://phpclasses. UpperDesign.com/browse. html/package/130) 29 Date and time utility class (http://phpclasses. UpperDesign.com/browse. html/package/230) Database access class (http://phpclasses. UpperDesign.com/browse. html/package/231) XML Writer class (http: //phpclasses.UpperDesign. com/browse.html/package/ 250) SOAP server class (http: //phpclasses.UpperDesign. com/browse.html/package/ 251) File cache class (http: //phpclasses.UpperDesign. com/browse.html/package/ 313) I-KnowMulti-word key expres- sion indexingand searching technologyfor building knowl- edge bases || 30 5. Future of MetaL This part addresses near future initiatives and opportunities of development with MetaL. 31 5.1. Planned work The development of MetaL depends on my current needs. The original motivation was to develop Integral : a modular Web based integrated system. Development of Integral is suspended due to the lack of time after my day job. Work on MetaL is often motivated by needs of my current job. | 32 5.2. Planned features and modules 1. Complete bindings for other Web programming languages like Perl and Java. 2. Module for building GUI applications. 3. Module for generating documentation in other formats besides HTML. 4. Bindings for compiled languages like C: Heavy-MetaL project. | 33 5.3. Planned Meta-Meta-Programming modules 1. Module to assist the development of new compiler modules in pure MetaL. 2. Database Procedure Language 3. Data Cliping Language | 34 5.4. Going Open Source A project like MetaL has too much potential to be maintained just by one developer. MetaL was not developed to take commercial advantage of its potential. It makes complete sense to open the source of MetaL to other developers interested to cooperate. The proof of concept of MetaL is to generate an application in two or more languages and demonstrate that all implementations work indierently. At the time of this writing, that proof of concept is almost done. After that, MetaL will be turned into a full edged Open Source (http://www.opensource.org/) project. 35 The Open Source license will be Apache (http: //www.opensource.org/licenses/apachepl.html)/BSD (http://www.opensource.org/licenses/bsd-license.html) like. | 36 5.5. Opportunities for other developers Carry on the development of Integral and its modules. Rewrite the compiler and its modules in lower level language like C/C++ to optimize compilation speed. Develop MetaL IDE - Integrated Development Environment. Develop bindings for other languages. Develop programs to reverse-engineer source code written in traditional languages to translate them to MetaL. Participate in the process of standardization and documentation of MetaL. || 37 6. Conclusion This part presents nal remarks and a pointer to obtain further information. 38 6.1. Final remarks MetaL exists now and has been used successfully in real world projects. It is not intended to replace the traditional programming languages. It is meant enhance the software development processes and improve the productivity of the developers. MetaL has indeed a lot of potential. It is a job for much more than one developer to take full advantage of MetaL potential. | 39 6.2. For more information... More information on MetaL is available at: www.meta- language.net (http://www.meta-language.net/) || 40 The End Thank you! Manuel Lemos mlemos@acm.org 41