You might not know that second to Delphi, my favorite programming language is Python. Also, my favorite version control system, Mercurial, is written in Python, and I keep all my Delphi projects in Mercurial repositories. I use Git when I have to, such as when contributing to Jedi JCL and JVCL, but when given the choice, I use Mercurial.
As an interpreted dynamically typed language, Python is wonderful, and it fits in easily when you need a general purpose programming language, or a script, or a system level tool like Mercurial, build-system tool, or some kind of quick parsing or text manipulation system, or when building a web-application. When you need something that compiles to optimized non-interpreted code, you usually write that extension in C. Python also makes it easy to write extensions. No I do not recommend trying to write extensions in Pascal.
Having said all this about Python, you can even use my new little open source project if you are not a Python programmer.
The project hasn't got a fancy name yet, it's just a mercurial repository with a few python scripts in it yet, but I have plans for all of these tools, and these plans will eventually converge into a bit of a swiss-army knife tool. I am building this tool on my own time, outside my current job's work hours, but I plan to use all of these little scripts while I work.
The project is here: https://bitbucket.org/wpostma/wppythonscripts
Now I will introduce the scripts, what they do, and what they might do later, when I have time to improve them. The reason for open sourcing them is that I hope that other people who need what I'm trying to provide here will also contribute their own features to these scripts.
fixdfm.py : Fix invalid .DFM files that won't open in Delphi
Have you ever seen this error: OBJECT expected on line ....
If you are like me, and it happens to you at 5:30 PM on a Friday after you just did a giant merge, you probably are not very happy to see it. It means that Delphi's DFM parser doesn't like your DFM. If the line number is 23, like here, perhaps you can open the .dfm and find the problem quickly. What if the error is at line 5223? How fun is that? And what if the DFM opens just fine, but all or most of the the controls are invisible, and appear to be gone. How do you like it when that happens at 5:30 PM on a Friday? Well if you're me, it does happen, and you need a tool to sort it out.
Let's run it on the broken .dfm. It's a command line tool, so I run it like this:
C:\dev\myapp>fixdfm.py myform.dfm
Scanning but not saving changes. Provide -save parameter to make changes
1 : inherited Form1: TForm1
testInheritUnit1.dfm : 23 : Unexpected Property Located After Child Object
Tag = 3
myform.dfm lines could be removed: 1
Changes made to myform.dfm not saved.
---
1 files scanned.
Now let's fix the DFM automatically:
C:\dev\myapp>fixdfm.py -save myform.dfm
Changes will be saved.
1 : inherited Form1: TForm1
myform.dfm : 23 : Unexpected Property Located After Child Object
Tag = 3
myform.dfm lines removed: 1
---
1 files scanned.
So, for some common cases, like when Merging just inserts some random property that should have belonged to a DFM object declaration in a place where no property declarations are allowed, Delphi does not offer to remove it for you, it just fails to open the DFM at all. What about if your merge tool removed an end keyword, or added an extra one? This tool cannot (yet) fix that, but it can detect it, and it can give you a map of where the dangling "scope" in the DFM is, so you can hopefully repair the damage.
A future version of this tool, will use Mercurial's version control system to permit you to scan DFMs, find broken ones, and then revert back through mercurial's log, one revision at a time, until it finds a non-broken DFM. Then it will repair the dfm, and rename the broken one to myform.dfm.orig so you can use a diff/merge tool to find any extra objects that should be added to the form, and add them yourself. And if I can, I will even make that part automatic. (Take DFM one, find objects that are in it that are not in DFM two, and shove them into the correct places in DFM two.)
dailyzip.py
This script is just to make a zip file containing the binary output of a build, or alternatively, a snapshot of source code of a project so you can have a snapshot that corresponds with your releases. It uses Mercurial to find the current repository hexadecimal ID, so you can always associate a daily zip back to the exact Repository revision that created it. The binary zip of all executables, DLLs and other outputs of a full build could be useful to your Beta testers or QA people. For any software product, a downloadable binary build snapshot showing the latest source still builds, and still works, can be very useful. Knowing what exact source code state corresponds to those binaries makes it even better. Since those hex ID codes from Mercurial are guaranteed unique, you will always know what goes with what.
Note that there is a flaw in my statement above. Knowing the tip revision is enough for most people, but some people might wonder "what bugs were fixed or features added between this daily build and the last". Such an auto-readme function could easily be added to dailyzip.py. Of course first I have to build a readme-builder.py script, which will extract a bug list like this:
BUG 1234 : WP : Fixed the thing that was
broken in the Thing. By the other thing.
Not that thing. The other one, in Unit3.pas.
As you can see, I am a fan of clear and unambiguous commit comments, which correspond to one and only one bug fix or atomic buildable change per commit. Always leave your repository buildable. Never commit unrelated changes together. And everybody will win.
getrepohex.py
This script is for grabbing the current repository revision (if the current folder is inside a Mercurial repository only) and doing something with it, like putting it into a file "ver.pas" or "ver.inc" that you could compile into or include into your Delphi app.
md5sum.py
I know there are a million MD5 hash calculation utilities out there, but I prefer to have mine in script form so I can use it to fingerprint things, and maybe even do a little further magic. By having most of what I need in a script instead of a binary, I find that when I need to do something like build a snapshot of a whole bunch of MD5 fingerprints, this script can serve as a basis for whatever task I need to do. Usually it involves knowing if the files I uploaded to FTP and down onto a client computer are binary equal (and not corrupted) from the time they left my original location, to wherever I'm getting them down to.
vertool.py
This little tool lets you extract version information from a Delphi .dproj file, or modify the version information
in a delphi .dproj file. If you ever find keeping a set of related executables and their projects all consistently at the same product revision level, you might find this useful.
Where is all this going?
A. Eventually I'll come up with a better name than wppythonscripts. The name I need would mean "These are mercurial/delphi code workflow and build automation helper utilities".
B. Eventually I will have a bit of a build and test automation framework put together that can build programs, unit test them, report pass/fail, build the installer MSI, and if it passes, it would be either upload it to an FTP site, or copied to a folder, for further QA/Testing or release processes to occur.
C. It will all play nice with existing tools like Hudson, and CruiseControl, and help take the output of running MSBUILD and make both nice build logs, and error reports, as well as help with statistics calculations so I can see the trends of error and warning counts, commits, builds, etc, from either Hudson or CruiseControl.
D. Because it is written in Python and targets Mercurial users, it is going to support some commit-hooks, which will prevent checkin of broken DFMs, prevent checkins of worthless noise DFM changes -- the Delphi form designer modifies and regenerates with meaningless changes to DFMs even when you just open and modify the .pas file. A meaningless change on two branches now means a meaningless and time-wasting merge conflict later, or even possibly, a corrupted .DFM. So the dfmfix tool will stop the insanity before it starts, it will soon be able to detect and revert such changes and auto-revert them before you can even commit something like this:
What is this not going to be?
A. Not a complete build tool. Use DANT or TRAIN or FinalBuilder.
B. Not a continuous integration tool. Use Jenkins/Hudson, CruiseControl, or FinalBuilder
C. Not something with a GUI, these are command line tools.
D. While I would love to build an equivalent to NuGet (Visual Studio), Maven (java), or CPAN (perl), I doubt I can accomplish that in this project.
If someone wants a GUI for the fixdfm.py, I'll accept a contribution of one, if it's coded in Python using the built in python tkinter GUI toolkit. Or I'll build you one, if you send me some money or a flat of nice Belgian beer.
This is an open source project, and it aims to make your life, if you are a Delphi developer who users Mercurial, a little nicer.
Please report bugs, especially please send me .DFM files mangled by your version control system. Any version control system that has to merge .DFMs is going to mangle them, if you have more than one developer opening and changing DFMs. Not only does Delphi make random looking order and object persistence property set changes, it also tends to reorder, remove, and add a lot of noise, especially if you have a codebase that started out in Delphi 5 through 7 and has survived a move up through several versions after that, perhaps up to XE2 or XE4. Please file bugs on the BitBucket site, and feature requests. I particularly find that the TMS components with their backwards-compatibility property-upgrade features, can cause some strange surprising things when opening and saving a DFM to make a change in a completely different area.
I really must recommend that all Delphi developers on ALL version control systems review and revert all .DFM changes that they did not intentionally make, to keep their version control history clean. But since asking all Delphi developers to do that is placing a burden on them, I think that a little help mapping Delphi's crazy .DFM changing tendencies to a stable version control system would be good.
As an interpreted dynamically typed language, Python is wonderful, and it fits in easily when you need a general purpose programming language, or a script, or a system level tool like Mercurial, build-system tool, or some kind of quick parsing or text manipulation system, or when building a web-application. When you need something that compiles to optimized non-interpreted code, you usually write that extension in C. Python also makes it easy to write extensions. No I do not recommend trying to write extensions in Pascal.
Having said all this about Python, you can even use my new little open source project if you are not a Python programmer.
The project hasn't got a fancy name yet, it's just a mercurial repository with a few python scripts in it yet, but I have plans for all of these tools, and these plans will eventually converge into a bit of a swiss-army knife tool. I am building this tool on my own time, outside my current job's work hours, but I plan to use all of these little scripts while I work.
The project is here: https://bitbucket.org/wpostma/wppythonscripts
Now I will introduce the scripts, what they do, and what they might do later, when I have time to improve them. The reason for open sourcing them is that I hope that other people who need what I'm trying to provide here will also contribute their own features to these scripts.
fixdfm.py : Fix invalid .DFM files that won't open in Delphi
Have you ever seen this error: OBJECT expected on line ....
If you are like me, and it happens to you at 5:30 PM on a Friday after you just did a giant merge, you probably are not very happy to see it. It means that Delphi's DFM parser doesn't like your DFM. If the line number is 23, like here, perhaps you can open the .dfm and find the problem quickly. What if the error is at line 5223? How fun is that? And what if the DFM opens just fine, but all or most of the the controls are invisible, and appear to be gone. How do you like it when that happens at 5:30 PM on a Friday? Well if you're me, it does happen, and you need a tool to sort it out.
C:\dev\myapp>fixdfm.py myform.dfm
Scanning but not saving changes. Provide -save parameter to make changes
1 : inherited Form1: TForm1
testInheritUnit1.dfm : 23 : Unexpected Property Located After Child Object
Tag = 3
myform.dfm lines could be removed: 1
Changes made to myform.dfm not saved.
---
1 files scanned.
Now let's fix the DFM automatically:
C:\dev\myapp>fixdfm.py -save myform.dfm
Changes will be saved.
1 : inherited Form1: TForm1
myform.dfm : 23 : Unexpected Property Located After Child Object
Tag = 3
myform.dfm lines removed: 1
---
1 files scanned.
So, for some common cases, like when Merging just inserts some random property that should have belonged to a DFM object declaration in a place where no property declarations are allowed, Delphi does not offer to remove it for you, it just fails to open the DFM at all. What about if your merge tool removed an end keyword, or added an extra one? This tool cannot (yet) fix that, but it can detect it, and it can give you a map of where the dangling "scope" in the DFM is, so you can hopefully repair the damage.
A future version of this tool, will use Mercurial's version control system to permit you to scan DFMs, find broken ones, and then revert back through mercurial's log, one revision at a time, until it finds a non-broken DFM. Then it will repair the dfm, and rename the broken one to myform.dfm.orig so you can use a diff/merge tool to find any extra objects that should be added to the form, and add them yourself. And if I can, I will even make that part automatic. (Take DFM one, find objects that are in it that are not in DFM two, and shove them into the correct places in DFM two.)
dailyzip.py
This script is just to make a zip file containing the binary output of a build, or alternatively, a snapshot of source code of a project so you can have a snapshot that corresponds with your releases. It uses Mercurial to find the current repository hexadecimal ID, so you can always associate a daily zip back to the exact Repository revision that created it. The binary zip of all executables, DLLs and other outputs of a full build could be useful to your Beta testers or QA people. For any software product, a downloadable binary build snapshot showing the latest source still builds, and still works, can be very useful. Knowing what exact source code state corresponds to those binaries makes it even better. Since those hex ID codes from Mercurial are guaranteed unique, you will always know what goes with what.
Note that there is a flaw in my statement above. Knowing the tip revision is enough for most people, but some people might wonder "what bugs were fixed or features added between this daily build and the last". Such an auto-readme function could easily be added to dailyzip.py. Of course first I have to build a readme-builder.py script, which will extract a bug list like this:
BUG 1234 : WP : Fixed the thing that was
broken in the Thing. By the other thing.
Not that thing. The other one, in Unit3.pas.
As you can see, I am a fan of clear and unambiguous commit comments, which correspond to one and only one bug fix or atomic buildable change per commit. Always leave your repository buildable. Never commit unrelated changes together. And everybody will win.
getrepohex.py
This script is for grabbing the current repository revision (if the current folder is inside a Mercurial repository only) and doing something with it, like putting it into a file "ver.pas" or "ver.inc" that you could compile into or include into your Delphi app.
md5sum.py
I know there are a million MD5 hash calculation utilities out there, but I prefer to have mine in script form so I can use it to fingerprint things, and maybe even do a little further magic. By having most of what I need in a script instead of a binary, I find that when I need to do something like build a snapshot of a whole bunch of MD5 fingerprints, this script can serve as a basis for whatever task I need to do. Usually it involves knowing if the files I uploaded to FTP and down onto a client computer are binary equal (and not corrupted) from the time they left my original location, to wherever I'm getting them down to.
vertool.py
This little tool lets you extract version information from a Delphi .dproj file, or modify the version information
in a delphi .dproj file. If you ever find keeping a set of related executables and their projects all consistently at the same product revision level, you might find this useful.
Where is all this going?
A. Eventually I'll come up with a better name than wppythonscripts. The name I need would mean "These are mercurial/delphi code workflow and build automation helper utilities".
B. Eventually I will have a bit of a build and test automation framework put together that can build programs, unit test them, report pass/fail, build the installer MSI, and if it passes, it would be either upload it to an FTP site, or copied to a folder, for further QA/Testing or release processes to occur.
C. It will all play nice with existing tools like Hudson, and CruiseControl, and help take the output of running MSBUILD and make both nice build logs, and error reports, as well as help with statistics calculations so I can see the trends of error and warning counts, commits, builds, etc, from either Hudson or CruiseControl.
D. Because it is written in Python and targets Mercurial users, it is going to support some commit-hooks, which will prevent checkin of broken DFMs, prevent checkins of worthless noise DFM changes -- the Delphi form designer modifies and regenerates with meaningless changes to DFMs even when you just open and modify the .pas file. A meaningless change on two branches now means a meaningless and time-wasting merge conflict later, or even possibly, a corrupted .DFM. So the dfmfix tool will stop the insanity before it starts, it will soon be able to detect and revert such changes and auto-revert them before you can even commit something like this:
What is this not going to be?
A. Not a complete build tool. Use DANT or TRAIN or FinalBuilder.
B. Not a continuous integration tool. Use Jenkins/Hudson, CruiseControl, or FinalBuilder
C. Not something with a GUI, these are command line tools.
D. While I would love to build an equivalent to NuGet (Visual Studio), Maven (java), or CPAN (perl), I doubt I can accomplish that in this project.
If someone wants a GUI for the fixdfm.py, I'll accept a contribution of one, if it's coded in Python using the built in python tkinter GUI toolkit. Or I'll build you one, if you send me some money or a flat of nice Belgian beer.
This is an open source project, and it aims to make your life, if you are a Delphi developer who users Mercurial, a little nicer.
Please report bugs, especially please send me .DFM files mangled by your version control system. Any version control system that has to merge .DFMs is going to mangle them, if you have more than one developer opening and changing DFMs. Not only does Delphi make random looking order and object persistence property set changes, it also tends to reorder, remove, and add a lot of noise, especially if you have a codebase that started out in Delphi 5 through 7 and has survived a move up through several versions after that, perhaps up to XE2 or XE4. Please file bugs on the BitBucket site, and feature requests. I particularly find that the TMS components with their backwards-compatibility property-upgrade features, can cause some strange surprising things when opening and saving a DFM to make a change in a completely different area.
I really must recommend that all Delphi developers on ALL version control systems review and revert all .DFM changes that they did not intentionally make, to keep their version control history clean. But since asking all Delphi developers to do that is placing a burden on them, I think that a little help mapping Delphi's crazy .DFM changing tendencies to a stable version control system would be good.
>You might not know that second to Delphi, my favorite
ReplyDelete>programming language is Python.
I know this; I just couldn't explain the order of those preferences if my life depended on it. :-)
>Also, my favorite version control system, Mercurial,
>is written in Python,
It's rather impressive nowadays all the places python is popping up.
>As a scripting language, Python is wonderful, and it
>fits in easily when you need a general purpose build-
>system tool, or some kind of quick parsing or text
>manipulation system, or when building a
>web-application.
I'd remove the word "scripting", place the period after "wonderful" and delete the rest of the paragraph. ;-) Seriously it's far outgrown being a tool for small scripts or a glue language; it's dominating the scientific field right now and dueling with Matlab and R for the mathematics field. It's also heavily used in quantitative finance along with R. DARPA has been looking for the ability to process large amounts of sensor and satellite data, found Python very favorable for the task and is distributing three million dollars in grant money to Continuum Analytics for "developing new techniques for data analysis and for visually portraying large, multi-dimensional data sets. The work aims to extend beyond the capabilities offered by the NumPy and SciPy Python libraries, which are widely used by programmers for mathematical and scientific calculations, respectively". Continuum's founder also created the Numba JIT compiler for Python.
All that said, it looks like you've put together an interesting collection of tools. It's something that I'd definitely consider contributing to in the future, if only to make Mason Wheeler's head explode over the idea that python is being used to fix Delphi. :-) :-)
@alcade: That was an imprecise choice of words. I should have said "an interpreted language". Python is truly interpreted based on a bytecode interpreter, and performance of that interpreter is so low compared with Pascal and C compiler users, that we are used to denigrating it as a "Scripting" language, however, as you say, it is suitable for something much larger than a scripting system, IPython and Mercurial, and TortoiseHG are enough proof of that. I would say it is a new kind of language, suitable for high level Applications Programming of all kinds, if you don't mind leaving your unobscured source code out in the open, it's about perfect. Attempts to turn Python into a single-executable-delivery vehicle are silly though. And so there's that Scripting word thrown about again. For closed source binary compiled commercial Win32 programming tasks, Python is still a toy, compared to Delphi. Thus the ordering.
ReplyDeleteRegarding Mercurial:
ReplyDeleteWe tried it when we wanted to get rid of CVS. But we did not find a good solution to run it as service on a Windows Server, so we switched to SVN instead. Even today I find no setup routine as it of course exists for SVN.
(I just found an installation description for using it with IIS, so there seems to be a possibility though...)
@jaenicke ; Did you even google it? :-) It took me 1 hour to set up a Mercurial server using Apache on windows, from not knowing how to do it at all, having one. Note that on Linux it takes about 10 seconds to set up a Mercurial server. IIS and Python are not a great match, it's far better to install a 20 meg Apache install than to use the gross mess that is IIS.
ReplyDeleteHere is the method I used:
ReplyDeletehttp://answers.bitnami.com/questions/5591/how-can-i-install-the-mercurial-plugin-into-the-bitnami-trac-stack
Hi, could you allow filing bug reports on bitbucket? There is no public issue tracker available currently.
ReplyDeleteHave you thought of using a Delphi DFM as the basis of a form for Python? At the simplest level, the Python could read the DFM and use Tkinter to create a Windows app. I'm using Python a lot to get quick answers to engineering problems but then I miss the RAD Delphi interface and amazing form tools. I don't know, maybe mixing code is just over complicating things but where you you find a scipy.optimize.fsolve for Delphi!
ReplyDelete